Sep 17 18:37:16.165545 ip-10-0-129-204 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 18:37:16.165557 ip-10-0-129-204 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 18:37:16.165564 ip-10-0-129-204 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 18:37:16.165754 ip-10-0-129-204 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 18:37:26.352648 ip-10-0-129-204 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 18:37:26.352665 ip-10-0-129-204 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ee6a6b8a03c6444d9ebc2eaaa88660e4 --
Sep 17 18:39:37.357143 ip-10-0-129-204 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:39:41.211897 ip-10-0-129-204 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:39:41.211897 ip-10-0-129-204 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:39:41.211897 ip-10-0-129-204 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:39:41.211897 ip-10-0-129-204 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:39:41.211897 ip-10-0-129-204 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:39:41.212611 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.212474    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:39:41.216297 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216279    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:41.216297 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216296    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:41.216297 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216299    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216304    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216307    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216309    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216312    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216315    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216318    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216320    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216323    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216325    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216334    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216337    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216339    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216342    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216344    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216347    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216350    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216353    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216355    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216358    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:41.216394 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216360    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216364    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216368    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216371    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216374    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216377    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216379    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216382    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216384    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216388    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216390    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216393    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216396    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216399    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216401    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216404    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216408    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216412    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216415    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:41.216902 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216418    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216420    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216423    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216425    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216428    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216430    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216432    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216435    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216437    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216440    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216442    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216445    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216447    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216449    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216452    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216455    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216457    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216460    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216462    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:41.217375 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216465    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216467    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216470    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216472    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216475    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216478    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216482    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216484    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216489    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216492    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216495    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216498    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216501    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216504    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216506    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216509    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216511    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216513    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216516    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216518    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:41.217829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216521    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216523    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216526    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216528    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216531    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216533    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216535    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216538    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216541    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216544    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216547    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216549    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216552    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216554    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216557    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216560    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216562    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216566    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.216569    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217333    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:41.218323 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217341    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217345    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217349    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217352    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217355    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217357    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217360    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217363    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217365    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217368    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217370    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217372    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217375    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217378    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217380    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217383    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217386    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217388    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217391    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217393    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:41.218804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217396    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217399    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217401    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217404    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217407    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217409    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217411    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217414    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217416    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217418    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217422    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217424    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217427    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217429    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217432    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217434    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217436    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217439    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217441    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217443    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:41.219359 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217446    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217448    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217450    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217454    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217458    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217461    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217463    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217466    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217468    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217471    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217473    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217476    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217478    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217481    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217484    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217487    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217489    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217492    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217494    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:41.219876 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217497    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217499    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217502    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217504    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217508    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217510    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217513    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217515    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217517    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217520    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217522    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217525    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217528    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217530    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217533    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217535    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217538    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217541    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217543    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217546    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:41.220346 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217549    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217552    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217555    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217557    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217560    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217562    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217566    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217570    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217573    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217576    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217579    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217582    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217584    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217587    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217590    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217593    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217595    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217599    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.217602    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:41.220854 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218015    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218021    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218024    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218027    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218030    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218033    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218036    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218039    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218042    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218044    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218050    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218052    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218055    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218057    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218060    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218063    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218065    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218068    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218071    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218073    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:41.221344 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218076    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218079    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218081    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218084    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218087    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218089    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218093    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218097    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218099    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218102    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218105    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218107    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218110    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218112    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218114    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218117    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218119    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218121    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218124    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218126    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:41.221842 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218129    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218131    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218135    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218137    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218141    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218143    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218146    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218150    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218152    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218154    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218157    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218160    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218162    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218165    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218167    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218170    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218172    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218175    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218177    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218180    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:41.222337 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218182    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218185    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218187    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218195    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218197    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218200    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218204    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218207    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218209    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218212    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218215    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218218    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218220    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218223    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218227    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218229    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218231    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218234    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218236    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:41.222884 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218240    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218243    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218251    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218253    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218256    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218258    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218262    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218264    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218267    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218271    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218274    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218277    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218280    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218283    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218285    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218288    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218291    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218295    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218298    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.218300    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:41.223342 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218371    2586 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218405    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218411    2586 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218415    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218420    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218423    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218427    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218432    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218435    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218438    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218441    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218444    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218448    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218451    2586 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218454    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218458    2586 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218460    2586 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218463    2586 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218468    2586 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218470    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218473    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218476    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218480    2586 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218483    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:39:41.223829 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218486    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218489    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218492    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218495    2586 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218509    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218513    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218518    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218521    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218525    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218528    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218531    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218533    2586 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218536    2586 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218539    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218543    2586 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218547    2586 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218550    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218553    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218557    2586 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218560    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218564    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218568    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218571    2586 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218573    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218576    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:39:41.224458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218579    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218582    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218585    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218588    2586 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218591    2586 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218594    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218597    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218600    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218603    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218606    2586 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218609    2586 flags.go:64] FLAG: --help="false"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218612    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218615    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218618    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218624    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218627    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218631    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218633    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218636    2586 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218639    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218642    2586 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218645    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218650    2586 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218653    2586 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:39:41.225140 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218656    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218658    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218661    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218665    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218668    2586 flags.go:64] FLAG: --lock-file=""
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218671    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218674    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218677    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218682    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218685    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218688    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218691    2586 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218694    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218698    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218701    2586 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218704    2586 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218708    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218711    2586 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218715    2586 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218718    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218721    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218723    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218726    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218730    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218733    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:39:41.225790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218736    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218743    2586 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218746    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218749    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218752    2586 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218756    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218762    2586 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218764    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218768    2586 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218771    2586 flags.go:64] FLAG: --port="10250"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218774    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218778    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0edd62edfa38a77f6"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218781    2586 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218784    2586 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218787    2586 flags.go:64] FLAG: --register-node="true"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218790    2586 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218794    2586 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218797    2586 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218799    2586 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218802    2586 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218806    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218809    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218812    2586 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218816    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218819    2586 flags.go:64] FLAG: --runonce="false"
Sep 17 18:39:41.226750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218821    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218824    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218827    2586 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218844    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218847    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218850    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218854    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218857    2586 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218860    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218863    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218866    2586 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218869    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218872    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218875    2586 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218878    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218884    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218887    2586 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218889    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218896    2586 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218898    2586 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218901    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218904    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218907    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218910    2586 flags.go:64] FLAG: --v="2"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218914    2586 flags.go:64] FLAG: --version="false"
Sep 17 18:39:41.227820 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218918    2586 flags.go:64] FLAG: --vmodule=""
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218922    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.218925    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219034    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219037    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219040    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219044    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219047    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219049    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219052    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219055    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219057    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219060    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219062    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219065    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219068    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219070    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219073    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219077    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:41.228959 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219081    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219083    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219086    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219089    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219091    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219094    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219100    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219103    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219106    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219108    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219111    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219114    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219116    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219119    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219121    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219124    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219127    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219129    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219132    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219134    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:41.229636 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219136    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219139    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219141    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219144    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219146    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219149    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219152    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219154    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219157    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219160    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219164    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219167    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219170    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219172    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219175    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219178    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219180    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219183    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219186    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219189    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:41.230364 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219192    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219194    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219197    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219199    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219201    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219204    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219207    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219209    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219212    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219214    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219217    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219219    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219222    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219224    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219227    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219229    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219232    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219234    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219238    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:41.231239 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219242    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219245    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219248    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219251    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219254    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219257    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219259    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219262    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219264    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219267    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219269    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219273    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219275    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219278    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219280    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219284    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219286    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219288    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219291    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219294    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:41.231904 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219296    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219299    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219301    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.219304    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.219311    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.227878    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.227906    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.227934    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.227944    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228031    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228039    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228044    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228049    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228053    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228057    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:41.232545 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228062    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228066    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228070    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228074    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228078    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228082    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228087    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228091    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228095    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228100    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228104    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228108    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228112    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228116    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228120    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228124    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228129    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228133    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228137    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228141    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:41.232979 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228145    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228156    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228160    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228166    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228170    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228174    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228179    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228183    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228188    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228192    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228196    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228200    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228204    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228208    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228212    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228216    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228220    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228225    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228229    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:41.233527 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228233    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228237    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228241    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228247    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228254    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228259    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228264    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228269    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228274    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228279    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228284    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228288    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228292    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228296    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228301    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228305    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228310    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228315    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228319    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:41.234106 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228324    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228328    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228332    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228336    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228340    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228345    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228350    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228354    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228358    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228362    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228367    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228371    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228375    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228380    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228384    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228388    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228392    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228396    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228401    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228405    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:41.234578 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228410    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228414    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228418    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228423    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228429    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228435    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228440    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228444    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228448    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228455    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228460    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228465    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228470    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228474    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:41.235124 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228479    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.228486    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228685    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228694    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228699    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228704    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228708    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228713    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228717    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228722    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228726    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228730    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228734    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228738    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228744    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:41.235503 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228748    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228752    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228756    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228762    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228766    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228770    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228774    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228778    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228782    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228786    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228790    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228794    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228798    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228803    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228807    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228811    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228815    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228819    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228823    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228827    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:41.235929 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228858    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228863    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228866    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228870    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228874    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228878    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228882    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228886    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228890    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228894    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228898    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228902    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228907    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228912    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228915    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228920    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228925    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228929    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228934    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228938    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228943    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:41.236414 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228947    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228951    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228955    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228959    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228963    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228968    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228972    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228978    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228984    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228990    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228995    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.228999    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229004    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229010    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229016    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229021    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229026    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229030    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:41.237125 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229035    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229040    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229045    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229052    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229057    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229061    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229067    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229071    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229075    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229079    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229083    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229087    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229092    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229096    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229102    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229106    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229111    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229114    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229118    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229123    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:41.237573 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229127    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229131    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229135    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229139    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229143    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229148    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:41.229151    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.229160    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.229326    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.232128    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.233145    2586 server.go:1019] "Starting client certificate rotation"
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.233251    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.233292    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:39:41.238073 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.236134    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:39:41.238399 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.237397    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:39:41.241730 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.241711    2586 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:39:41.243425 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.243410    2586 log.go:25] "Validated CRI v1 image API"
Sep 17 18:39:41.244195 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.244175    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:39:41.253449 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.253425    2586 fs.go:135] Filesystem UUIDs: map[4d33acb9-05bc-4cea-97d6-e37585ed894e:/dev/nvme0n1p3 5b006f4f-3a19-444d-a318-72de4045effe:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 18:39:41.253530 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.253447    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:39:41.259299 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.259171    2586 manager.go:216] Machine: {Timestamp:2026-09-17 18:39:41.258655894 +0000 UTC m=+2.118480555 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3060085 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec204ba2a15a313573e932ee67204b0e SystemUUID:ec204ba2-a15a-3135-73e9-32ee67204b0e BootID:ee6a6b8a-03c6-444d-9ebc-2eaaa88660e4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:33:bc:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:33:bc:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:08:a8:87:4b:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:39:41.259299 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.259286    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:39:41.259437 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.259405    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:39:41.261043 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.261019    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:39:41.261208 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.261046    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-204.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:39:41.262307 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.262296    2586 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:39:41.262338 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.262311    2586 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:39:41.262338 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.262319    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:39:41.262338 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.262331    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:39:41.263550 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.263539    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:39:41.267317 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.267306    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:39:41.267442 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.267433    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:39:41.267478 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.267469    2586 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:39:41.267508 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.267486    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:39:41.267539 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.267511    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:39:41.267539 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.267525    2586 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:39:41.267539 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.267537    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:39:41.267992 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.267980    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:39:41.268047 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.267999    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:39:41.270229 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.270209    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:39:41.272760 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.272747    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:39:41.272814 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.272777    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:39:41.273989 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.273976    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:39:41.274069 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.273996    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:39:41.274069 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274006    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:39:41.274069 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274012    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:39:41.274069 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274021    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:39:41.274069 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274026    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:39:41.274069 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274039    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:39:41.274069 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274044    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:39:41.274069 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274052    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:39:41.274069 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274061    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:39:41.274362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274078    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:39:41.274362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274085    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:39:41.274362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274109    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:39:41.274362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.274114    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:39:41.276201 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.276182    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:39:41.278106 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.278091    2586 server.go:1268] "Started kubelet"
Sep 17 18:39:41.278202 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.278162    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:39:41.278292 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.278246    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:39:41.278336 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.278317    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:39:41.279383 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.279360    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:39:41.279775 ip-10-0-129-204 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:39:41.280462 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.280450    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-204.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:39:41.280525 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.280469    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:39:41.280593 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.280571    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:39:41.280681 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.280670    2586 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:39:41.283188 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.283165    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:39:41.288496 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.288482    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:39:41.288576 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.288501    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:39:41.288637 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.288589    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:41.289206 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289189    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:39:41.289343 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289190    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:39:41.289343 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289224    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:39:41.289343 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289289    2586 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:39:41.289343 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289299    2586 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:39:41.289545 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289386    2586 factory.go:55] Registering systemd factory
Sep 17 18:39:41.289545 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289431    2586 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:39:41.289671 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289645    2586 factory.go:153] Registering CRI-O factory
Sep 17 18:39:41.289671 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289657    2586 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:39:41.289733 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289722    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:39:41.289767 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289739    2586 factory.go:103] Registering Raw factory
Sep 17 18:39:41.289767 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.289751    2586 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:39:41.290387 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.290373    2586 manager.go:349] Starting recovery of all containers
Sep 17 18:39:41.290567 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.290474    2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:39:41.290695 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.290647    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:39:41.322627 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.322602    2586 manager.go:354] Recovery completed
Sep 17 18:39:41.326527 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.326510    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:41.328678 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.328653    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:41.328773 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.328695    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:41.328773 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.328705    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:41.329259 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.329245    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:39:41.329259 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.329259    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:41.329403 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.329258    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:39:41.329403 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.329323    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:39:41.329530 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.328552    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-204.ec2.internal.18d62fbacd1ddc6b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-204.ec2.internal,UID:ip-10-0-129-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-204.ec2.internal,},FirstTimestamp:2026-09-17 18:39:41.278059627 +0000 UTC m=+2.137884265,LastTimestamp:2026-09-17 18:39:41.278059627 +0000 UTC m=+2.137884265,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-204.ec2.internal,}"
Sep 17 18:39:41.330966 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.330414    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:41.330966 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.330441    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:41.330966 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.330455    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:41.332094 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.332081    2586 policy_none.go:49] "None policy: Start"
Sep 17 18:39:41.332135 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.332099    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:39:41.332135 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.332109    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:39:41.334053 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.334039    2586 policy_none.go:47] "Start"
Sep 17 18:39:41.339920 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.339854    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-204.ec2.internal.18d62fbad0223f25  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-204.ec2.internal,UID:ip-10-0-129-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-204.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-204.ec2.internal,},FirstTimestamp:2026-09-17 18:39:41.328678693 +0000 UTC m=+2.188503334,LastTimestamp:2026-09-17 18:39:41.328678693 +0000 UTC m=+2.188503334,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-204.ec2.internal,}"
Sep 17 18:39:41.346368 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.346307    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-204.ec2.internal.18d62fbad022919b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-204.ec2.internal,UID:ip-10-0-129-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-204.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-204.ec2.internal,},FirstTimestamp:2026-09-17 18:39:41.328699803 +0000 UTC m=+2.188524442,LastTimestamp:2026-09-17 18:39:41.328699803 +0000 UTC m=+2.188524442,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-204.ec2.internal,}"
Sep 17 18:39:41.353509 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.353444    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-204.ec2.internal.18d62fbad022b8f8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-204.ec2.internal,UID:ip-10-0-129-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-204.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-204.ec2.internal,},FirstTimestamp:2026-09-17 18:39:41.32870988 +0000 UTC m=+2.188534519,LastTimestamp:2026-09-17 18:39:41.32870988 +0000 UTC m=+2.188534519,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-204.ec2.internal,}"
Sep 17 18:39:41.356065 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.356046    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nshkx"
Sep 17 18:39:41.357954 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.357877    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-204.ec2.internal.18d62fbad0223f25\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-204.ec2.internal.18d62fbad0223f25  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-204.ec2.internal,UID:ip-10-0-129-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-204.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-204.ec2.internal,},FirstTimestamp:2026-09-17 18:39:41.328678693 +0000 UTC m=+2.188503334,LastTimestamp:2026-09-17 18:39:41.33042912 +0000 UTC m=+2.190253764,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-204.ec2.internal,}"
Sep 17 18:39:41.371978 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.371908    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-204.ec2.internal.18d62fbad022919b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-204.ec2.internal.18d62fbad022919b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-204.ec2.internal,UID:ip-10-0-129-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-204.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-204.ec2.internal,},FirstTimestamp:2026-09-17 18:39:41.328699803 +0000 UTC m=+2.188524442,LastTimestamp:2026-09-17 18:39:41.330446993 +0000 UTC m=+2.190271637,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-204.ec2.internal,}"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.374476    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nshkx"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.376513    2586 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.376552    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.376564    2586 server.go:87] "Starting device plugin registration server"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.376909    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.376929    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.377013    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.377100    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.377109    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.377706    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.377747    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:41.387055 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.382144    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-204.ec2.internal.18d62fbad022b8f8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-204.ec2.internal.18d62fbad022b8f8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-204.ec2.internal,UID:ip-10-0-129-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-204.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-204.ec2.internal,},FirstTimestamp:2026-09-17 18:39:41.32870988 +0000 UTC m=+2.188534519,LastTimestamp:2026-09-17 18:39:41.330459986 +0000 UTC m=+2.190284626,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-204.ec2.internal,}"
Sep 17 18:39:41.450354 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.450318    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:39:41.451862 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.451824    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:39:41.451975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.451899    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:39:41.451975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.451929    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:39:41.451975 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.451966    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:39:41.454348 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.454330    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:41.477595 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.477523    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:41.478459 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.478441    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:41.478569 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.478478    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:41.478569 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.478494    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:41.478569 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.478516    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.489789 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.489764    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.489789 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.489788    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-204.ec2.internal\": node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:41.517988 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.517957    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:41.552302 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.552277    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal"]
Sep 17 18:39:41.552390 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.552352    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:41.553348 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.553326    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:41.553446 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.553357    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:41.553446 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.553373    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:41.555541 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.555528    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:41.555702 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.555687    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.555752 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.555718    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:41.556286 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.556269    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:41.556382 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.556297    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:41.556382 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.556306    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:41.556382 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.556269    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:41.556523 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.556384    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:41.556523 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.556401    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:41.558292 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.558278    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.558354 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.558303    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:41.559012 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.558994    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:41.559092 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.559025    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:41.559092 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.559036    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:41.577419 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.577396    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-204.ec2.internal\" not found" node="ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.581274 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.581256    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-204.ec2.internal\" not found" node="ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.590812 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.590792    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc730dc409eabf98dcfae53f4dbd938e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"cc730dc409eabf98dcfae53f4dbd938e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.590927 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.590818    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc730dc409eabf98dcfae53f4dbd938e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"cc730dc409eabf98dcfae53f4dbd938e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.590927 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.590851    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/894af7d1d57be17aaf3ffac63d929c3b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-204.ec2.internal\" (UID: \"894af7d1d57be17aaf3ffac63d929c3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.618304 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.618280    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:41.691446 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.691414    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/894af7d1d57be17aaf3ffac63d929c3b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-204.ec2.internal\" (UID: \"894af7d1d57be17aaf3ffac63d929c3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.691446 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.691448    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc730dc409eabf98dcfae53f4dbd938e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"cc730dc409eabf98dcfae53f4dbd938e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.691614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.691464    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc730dc409eabf98dcfae53f4dbd938e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"cc730dc409eabf98dcfae53f4dbd938e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.691614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.691504    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc730dc409eabf98dcfae53f4dbd938e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"cc730dc409eabf98dcfae53f4dbd938e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.691614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.691514    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/894af7d1d57be17aaf3ffac63d929c3b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-204.ec2.internal\" (UID: \"894af7d1d57be17aaf3ffac63d929c3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.691614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.691541    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc730dc409eabf98dcfae53f4dbd938e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal\" (UID: \"cc730dc409eabf98dcfae53f4dbd938e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.718455 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.718407    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:41.819452 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.819377    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:41.881602 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.881573    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.884713 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:41.884693    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:41.919641 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:41.919610    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:42.020252 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:42.020202    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:42.120761 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:42.120678    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:42.170134 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.170103    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:42.221751 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:42.221719    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:42.232916 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.232882    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:39:42.233066 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.233048    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:39:42.233103 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.233069    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:39:42.322084 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:42.322049    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:42.380458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.380367    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:34:41 +0000 UTC" deadline="2028-02-22 10:55:59.493181247 +0000 UTC"
Sep 17 18:39:42.380458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.380419    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12544h16m17.112767492s"
Sep 17 18:39:42.423015 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:42.422855    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:42.429979 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.429956    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:42.432343 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.432324    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:42.432417 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.432359    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:42.432417 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.432372    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:42.523696 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:42.523667    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:42.532323 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.532296    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:42.624711 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:42.624681    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:42.634829 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:42.634767    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc730dc409eabf98dcfae53f4dbd938e.slice/crio-8f408fa5ed9bd64343fae03d68f0132da0d57ce86c97a95d24123b9ceb4cb6c5 WatchSource:0}: Error finding container 8f408fa5ed9bd64343fae03d68f0132da0d57ce86c97a95d24123b9ceb4cb6c5: Status 404 returned error can't find the container with id 8f408fa5ed9bd64343fae03d68f0132da0d57ce86c97a95d24123b9ceb4cb6c5
Sep 17 18:39:42.639581 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.639568    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:39:42.725614 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:42.725575    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-204.ec2.internal\" not found"
Sep 17 18:39:42.746040 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.746009    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:42.764690 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:42.764664    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894af7d1d57be17aaf3ffac63d929c3b.slice/crio-d358c1f5ce2b0345fb2ce33ccf8db9df8877c3825597ca346ca95689c2646f13 WatchSource:0}: Error finding container d358c1f5ce2b0345fb2ce33ccf8db9df8877c3825597ca346ca95689c2646f13: Status 404 returned error can't find the container with id d358c1f5ce2b0345fb2ce33ccf8db9df8877c3825597ca346ca95689c2646f13
Sep 17 18:39:42.789654 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.789628    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:42.799295 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.799275    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:39:42.800313 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.800301    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal"
Sep 17 18:39:42.813554 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:42.813529    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:39:43.268657 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.268623    2586 apiserver.go:52] "Watching apiserver"
Sep 17 18:39:43.276200 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.276175    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:39:43.276585 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.276564    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-dxrln","openshift-multus/multus-5lkxr","openshift-multus/network-metrics-daemon-vhljw","openshift-network-diagnostics/network-check-target-cc2x9","kube-system/konnectivity-agent-xwwg7","kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42","openshift-image-registry/node-ca-8mvc8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal","openshift-multus/multus-additional-cni-plugins-2bgpb","openshift-network-operator/iptables-alerter-6bb78","openshift-ovn-kubernetes/ovnkube-node-pjczz","kube-system/global-pull-secret-syncer-xnrlh","openshift-cluster-node-tuning-operator/tuned-jvtzp"]
Sep 17 18:39:43.279149 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.279123    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.281586 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.281345    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:43.281586 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.281411    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:39:43.281724 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.281689    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:39:43.282054 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.282036    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-889xn\""
Sep 17 18:39:43.282267 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.282254    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:39:43.282916 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.282902    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:39:43.283861 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.283451    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:43.283861 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.283519    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:39:43.286080 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.285986    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:43.286080 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.286046    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:39:43.287738 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.287721    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:39:43.289710 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.289685    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:39:43.290054 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.290040    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.290557 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.290531    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:39:43.290799 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.290782    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jj4dg\""
Sep 17 18:39:43.290887 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.290851    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:39:43.292817 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.292797    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.293234 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.293208    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xhnps\""
Sep 17 18:39:43.293351 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.293334    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:43.293625 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.293604    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:39:43.295456 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.295197    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.295589 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.295568    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:39:43.295908 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.295890    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:39:43.296337 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.296318    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:39:43.296422 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.296339    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6j4b8\""
Sep 17 18:39:43.298100 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.297452    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.298100 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.297534    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:39:43.298100 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.297537    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:39:43.298100 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.297979    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5zppv\""
Sep 17 18:39:43.299822 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.299806    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:39:43.299930 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.299919    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.300233 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300214    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:39:43.300322 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300214    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.300322 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300299    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-socket-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.300322 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300318    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:39:43.300467 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300329    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-registration-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.300467 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300362    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-systemd\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.300467 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300385    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-modprobe-d\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.300467 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300420    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-sysctl-d\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.300467 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300442    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-sysctl-conf\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.300467 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300466    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-host\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300486    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-sysconfig\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300502    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-var-lib-kubelet\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300517    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49b06cb0-be28-4d6f-9bef-694ed58104c8-tmp\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300532    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-device-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300551    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-sys-fs\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300573    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-tuned\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300587    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300604    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-run\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300624    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1463c73f-f522-4ea3-86f1-4ce25079b61c-agent-certs\") pod \"konnectivity-agent-xwwg7\" (UID: \"1463c73f-f522-4ea3-86f1-4ce25079b61c\") " pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300645    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf28n\" (UniqueName: \"kubernetes.io/projected/e8969506-84fa-4bcc-a5f6-383cbe2b041f-kube-api-access-pf28n\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:43.300695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300663    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300723    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1463c73f-f522-4ea3-86f1-4ce25079b61c-konnectivity-ca\") pod \"konnectivity-agent-xwwg7\" (UID: \"1463c73f-f522-4ea3-86f1-4ce25079b61c\") " pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300745    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300762    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9vw\" (UniqueName: \"kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw\") pod \"network-check-target-cc2x9\" (UID: \"19bca968-8c00-452e-93fb-8edd90e3f271\") " pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300779    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-sys\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300819    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-kubernetes\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300867    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-lib-modules\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300893    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csj2s\" (UniqueName: \"kubernetes.io/projected/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-kube-api-access-csj2s\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300928    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nqxpc\""
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300932    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/670d8bfc-6813-421f-941d-c7e35a57f000-kubelet-config\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300951    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.300986    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbcvh\" (UniqueName: \"kubernetes.io/projected/49b06cb0-be28-4d6f-9bef-694ed58104c8-kube-api-access-mbcvh\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.301175 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.301034    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/670d8bfc-6813-421f-941d-c7e35a57f000-dbus\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:43.301930 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.301911    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:39:43.302190 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.302172    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:39:43.302464 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.302445    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thh2h\""
Sep 17 18:39:43.302552 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.302488    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:39:43.302700 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.302681    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.305013 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.304994    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6tsk8\""
Sep 17 18:39:43.305157 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.305135    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:43.305245 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.305136    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:39:43.305301 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.305270    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.305607 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.305590    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:39:43.307456 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.307436    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:39:43.307548 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.307463    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qb2mk\""
Sep 17 18:39:43.308449 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.308435    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:39:43.308540 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.308476    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:39:43.309227 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.309206    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:39:43.309389 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.309364    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:39:43.309664 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.309560    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:39:43.322976 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.322955    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-84qbd"
Sep 17 18:39:43.333877 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.333828    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-84qbd"
Sep 17 18:39:43.390492 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.390459    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:39:43.401710 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401677    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-sysctl-conf\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.401710 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401718    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-host\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.401975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401749    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1cf4c16-7f5e-415b-848a-fef07ac8160b-ovnkube-config\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.401975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401773    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-device-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.401975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401800    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:43.401975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401826    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/228d79c5-beb7-4a41-b877-c68fae3efc15-system-cni-dir\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.401975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401869    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-cnibin\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.401975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401897    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1cf4c16-7f5e-415b-848a-fef07ac8160b-env-overrides\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.401975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401922    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1cf4c16-7f5e-415b-848a-fef07ac8160b-ovnkube-script-lib\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.401975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401945    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1cf4c16-7f5e-415b-848a-fef07ac8160b-ovn-node-metrics-cert\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.401975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401969    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf28n\" (UniqueName: \"kubernetes.io/projected/e8969506-84fa-4bcc-a5f6-383cbe2b041f-kube-api-access-pf28n\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.401992    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/228d79c5-beb7-4a41-b877-c68fae3efc15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402016    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-socket-dir-parent\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402038    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-hostroot\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402059    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-run-openvswitch\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402080    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-log-socket\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402104    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402137    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nvqs\" (UniqueName: \"kubernetes.io/projected/f1cf4c16-7f5e-415b-848a-fef07ac8160b-kube-api-access-8nvqs\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402163    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402186    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-run-netns\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402211    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-run-multus-certs\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402233    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-run-ovn\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402257    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-sys\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402281    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csj2s\" (UniqueName: \"kubernetes.io/projected/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-kube-api-access-csj2s\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402297    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/670d8bfc-6813-421f-941d-c7e35a57f000-kubelet-config\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402311    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/228d79c5-beb7-4a41-b877-c68fae3efc15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.402439 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402331    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-var-lib-kubelet\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402353    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-kubelet\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402383    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-run-netns\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402405    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6caa7aaf-fa9b-482d-b166-30b9f3515174-tmp-dir\") pod \"node-resolver-dxrln\" (UID: \"6caa7aaf-fa9b-482d-b166-30b9f3515174\") " pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402428    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx77m\" (UniqueName: \"kubernetes.io/projected/1af79fd6-6d59-4d91-bbee-226e76b02e32-kube-api-access-bx77m\") pod \"node-ca-8mvc8\" (UID: \"1af79fd6-6d59-4d91-bbee-226e76b02e32\") " pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402450    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb7d17e7-5508-41b4-83d5-e5130b662b81-cni-binary-copy\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402473    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-run-k8s-cni-cncf-io\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402495    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-conf-dir\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402520    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-run-systemd\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402535    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-etc-openvswitch\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402550    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6caa7aaf-fa9b-482d-b166-30b9f3515174-hosts-file\") pod \"node-resolver-dxrln\" (UID: \"6caa7aaf-fa9b-482d-b166-30b9f3515174\") " pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402565    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/228d79c5-beb7-4a41-b877-c68fae3efc15-cnibin\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402579    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-588k9\" (UniqueName: \"kubernetes.io/projected/228d79c5-beb7-4a41-b877-c68fae3efc15-kube-api-access-588k9\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402593    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa5fd544-ef17-4ad6-8a7d-d68afb85827a-iptables-alerter-script\") pod \"iptables-alerter-6bb78\" (UID: \"aa5fd544-ef17-4ad6-8a7d-d68afb85827a\") " pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402616    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa5fd544-ef17-4ad6-8a7d-d68afb85827a-host-slash\") pod \"iptables-alerter-6bb78\" (UID: \"aa5fd544-ef17-4ad6-8a7d-d68afb85827a\") " pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402641    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.403290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402670    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-socket-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402693    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-systemd\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402717    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-sysconfig\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402741    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-var-lib-kubelet\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402764    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49b06cb0-be28-4d6f-9bef-694ed58104c8-tmp\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402786    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-sys-fs\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402805    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/228d79c5-beb7-4a41-b877-c68fae3efc15-cni-binary-copy\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402827    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/228d79c5-beb7-4a41-b877-c68fae3efc15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402869    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-cni-dir\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402885    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-var-lib-cni-bin\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402909    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-run\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402937    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1af79fd6-6d59-4d91-bbee-226e76b02e32-host\") pod \"node-ca-8mvc8\" (UID: \"1af79fd6-6d59-4d91-bbee-226e76b02e32\") " pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402961    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-var-lib-cni-multus\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.402989    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-daemon-config\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403018    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-systemd-units\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403042    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-slash\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403067    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-var-lib-openvswitch\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.404082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403087    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1463c73f-f522-4ea3-86f1-4ce25079b61c-agent-certs\") pod \"konnectivity-agent-xwwg7\" (UID: \"1463c73f-f522-4ea3-86f1-4ce25079b61c\") " pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403103    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz67b\" (UniqueName: \"kubernetes.io/projected/aa5fd544-ef17-4ad6-8a7d-d68afb85827a-kube-api-access-gz67b\") pod \"iptables-alerter-6bb78\" (UID: \"aa5fd544-ef17-4ad6-8a7d-d68afb85827a\") " pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403119    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-sysctl-d\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403132    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-node-log\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403147    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-cni-bin\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403162    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-cni-netd\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403178    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1463c73f-f522-4ea3-86f1-4ce25079b61c-konnectivity-ca\") pod \"konnectivity-agent-xwwg7\" (UID: \"1463c73f-f522-4ea3-86f1-4ce25079b61c\") " pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403200    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403226    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qmh\" (UniqueName: \"kubernetes.io/projected/fb7d17e7-5508-41b4-83d5-e5130b662b81-kube-api-access-28qmh\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403259    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403287    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9vw\" (UniqueName: \"kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw\") pod \"network-check-target-cc2x9\" (UID: \"19bca968-8c00-452e-93fb-8edd90e3f271\") " pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403312    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-system-cni-dir\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403340    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-kubernetes\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403365    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-lib-modules\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403383    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-etc-kubernetes\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403400    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbcvh\" (UniqueName: \"kubernetes.io/projected/49b06cb0-be28-4d6f-9bef-694ed58104c8-kube-api-access-mbcvh\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403414    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/670d8bfc-6813-421f-941d-c7e35a57f000-dbus\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:43.404962 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403432    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9prg\" (UniqueName: \"kubernetes.io/projected/6caa7aaf-fa9b-482d-b166-30b9f3515174-kube-api-access-g9prg\") pod \"node-resolver-dxrln\" (UID: \"6caa7aaf-fa9b-482d-b166-30b9f3515174\") " pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403451    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-tuned\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403465    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1af79fd6-6d59-4d91-bbee-226e76b02e32-serviceca\") pod \"node-ca-8mvc8\" (UID: \"1af79fd6-6d59-4d91-bbee-226e76b02e32\") " pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403480    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-registration-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403497    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-modprobe-d\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403612    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-modprobe-d\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403718    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-sysctl-conf\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403763    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-host\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.403806    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-device-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.403912    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.403971    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret podName:670d8bfc-6813-421f-941d-c7e35a57f000 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:43.903947553 +0000 UTC m=+4.763772179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret") pod "global-pull-secret-syncer-xnrlh" (UID: "670d8bfc-6813-421f-941d-c7e35a57f000") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.404455    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.404509    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-sys\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.404665    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/670d8bfc-6813-421f-941d-c7e35a57f000-kubelet-config\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.404805    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.404938    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-socket-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.404992    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-systemd\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.405826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.405038    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-sysconfig\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.406695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.405082    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-var-lib-kubelet\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.406695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.405323    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:39:43.406695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.405368    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-sys-fs\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.406695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.405498    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-run\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.407110 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.407029    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-lib-modules\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.407110 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.407077    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/670d8bfc-6813-421f-941d-c7e35a57f000-dbus\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:43.407110 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.407080    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-sysctl-d\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.408035 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.407150    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-kubernetes\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.408035 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.407172    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:43.408035 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.407232    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs podName:e8969506-84fa-4bcc-a5f6-383cbe2b041f nodeName:}" failed. No retries permitted until 2026-09-17 18:39:43.907215152 +0000 UTC m=+4.767039794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs") pod "network-metrics-daemon-vhljw" (UID: "e8969506-84fa-4bcc-a5f6-383cbe2b041f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:43.408035 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.407343    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-registration-dir\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.408035 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.407775    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1463c73f-f522-4ea3-86f1-4ce25079b61c-konnectivity-ca\") pod \"konnectivity-agent-xwwg7\" (UID: \"1463c73f-f522-4ea3-86f1-4ce25079b61c\") " pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:39:43.409411 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.409366    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49b06cb0-be28-4d6f-9bef-694ed58104c8-tmp\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.409546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.409456    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49b06cb0-be28-4d6f-9bef-694ed58104c8-etc-tuned\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.409651 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.409631    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1463c73f-f522-4ea3-86f1-4ce25079b61c-agent-certs\") pod \"konnectivity-agent-xwwg7\" (UID: \"1463c73f-f522-4ea3-86f1-4ce25079b61c\") " pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:39:43.413291 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.413269    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:43.413402 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.413295    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:43.413402 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.413311    2586 projected.go:196] Error preparing data for projected volume kube-api-access-kd9vw for pod openshift-network-diagnostics/network-check-target-cc2x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:43.413402 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.413373    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw podName:19bca968-8c00-452e-93fb-8edd90e3f271 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:43.913356065 +0000 UTC m=+4.773180712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kd9vw" (UniqueName: "kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw") pod "network-check-target-cc2x9" (UID: "19bca968-8c00-452e-93fb-8edd90e3f271") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:43.416225 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.416145    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbcvh\" (UniqueName: \"kubernetes.io/projected/49b06cb0-be28-4d6f-9bef-694ed58104c8-kube-api-access-mbcvh\") pod \"tuned-jvtzp\" (UID: \"49b06cb0-be28-4d6f-9bef-694ed58104c8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.416327 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.416221    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csj2s\" (UniqueName: \"kubernetes.io/projected/beb890a7-3e1f-43fb-96c0-129d39c4f9b0-kube-api-access-csj2s\") pod \"aws-ebs-csi-driver-node-s8w42\" (UID: \"beb890a7-3e1f-43fb-96c0-129d39c4f9b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.416678 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.416657    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf28n\" (UniqueName: \"kubernetes.io/projected/e8969506-84fa-4bcc-a5f6-383cbe2b041f-kube-api-access-pf28n\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:43.457997 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.457937    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" event={"ID":"894af7d1d57be17aaf3ffac63d929c3b","Type":"ContainerStarted","Data":"d358c1f5ce2b0345fb2ce33ccf8db9df8877c3825597ca346ca95689c2646f13"}
Sep 17 18:39:43.459396 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.459365    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" event={"ID":"cc730dc409eabf98dcfae53f4dbd938e","Type":"ContainerStarted","Data":"8f408fa5ed9bd64343fae03d68f0132da0d57ce86c97a95d24123b9ceb4cb6c5"}
Sep 17 18:39:43.504554 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504505    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-etc-kubernetes\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.504554 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504544    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9prg\" (UniqueName: \"kubernetes.io/projected/6caa7aaf-fa9b-482d-b166-30b9f3515174-kube-api-access-g9prg\") pod \"node-resolver-dxrln\" (UID: \"6caa7aaf-fa9b-482d-b166-30b9f3515174\") " pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504570    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1af79fd6-6d59-4d91-bbee-226e76b02e32-serviceca\") pod \"node-ca-8mvc8\" (UID: \"1af79fd6-6d59-4d91-bbee-226e76b02e32\") " pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504589    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1cf4c16-7f5e-415b-848a-fef07ac8160b-ovnkube-config\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504614    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/228d79c5-beb7-4a41-b877-c68fae3efc15-system-cni-dir\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504631    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-cnibin\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504650    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1cf4c16-7f5e-415b-848a-fef07ac8160b-env-overrides\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504673    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1cf4c16-7f5e-415b-848a-fef07ac8160b-ovnkube-script-lib\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504690    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1cf4c16-7f5e-415b-848a-fef07ac8160b-ovn-node-metrics-cert\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504714    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/228d79c5-beb7-4a41-b877-c68fae3efc15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504737    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-socket-dir-parent\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504760    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-hostroot\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504782    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-run-openvswitch\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504804    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-log-socket\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504853    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.504888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504881    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nvqs\" (UniqueName: \"kubernetes.io/projected/f1cf4c16-7f5e-415b-848a-fef07ac8160b-kube-api-access-8nvqs\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504907    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-run-netns\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504931    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-run-multus-certs\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504953    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-run-ovn\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.504983    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/228d79c5-beb7-4a41-b877-c68fae3efc15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505004    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-var-lib-kubelet\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505018    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-kubelet\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505031    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-run-netns\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505058    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6caa7aaf-fa9b-482d-b166-30b9f3515174-tmp-dir\") pod \"node-resolver-dxrln\" (UID: \"6caa7aaf-fa9b-482d-b166-30b9f3515174\") " pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505073    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx77m\" (UniqueName: \"kubernetes.io/projected/1af79fd6-6d59-4d91-bbee-226e76b02e32-kube-api-access-bx77m\") pod \"node-ca-8mvc8\" (UID: \"1af79fd6-6d59-4d91-bbee-226e76b02e32\") " pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505106    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb7d17e7-5508-41b4-83d5-e5130b662b81-cni-binary-copy\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505123    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-run-k8s-cni-cncf-io\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505151    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-conf-dir\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505176    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-run-systemd\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505199    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-etc-openvswitch\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505224    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6caa7aaf-fa9b-482d-b166-30b9f3515174-hosts-file\") pod \"node-resolver-dxrln\" (UID: \"6caa7aaf-fa9b-482d-b166-30b9f3515174\") " pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505247    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/228d79c5-beb7-4a41-b877-c68fae3efc15-cnibin\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.505613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505273    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-588k9\" (UniqueName: \"kubernetes.io/projected/228d79c5-beb7-4a41-b877-c68fae3efc15-kube-api-access-588k9\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505298    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa5fd544-ef17-4ad6-8a7d-d68afb85827a-iptables-alerter-script\") pod \"iptables-alerter-6bb78\" (UID: \"aa5fd544-ef17-4ad6-8a7d-d68afb85827a\") " pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505324    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa5fd544-ef17-4ad6-8a7d-d68afb85827a-host-slash\") pod \"iptables-alerter-6bb78\" (UID: \"aa5fd544-ef17-4ad6-8a7d-d68afb85827a\") " pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505389    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa5fd544-ef17-4ad6-8a7d-d68afb85827a-host-slash\") pod \"iptables-alerter-6bb78\" (UID: \"aa5fd544-ef17-4ad6-8a7d-d68afb85827a\") " pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505440    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-run-netns\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505477    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-run-multus-certs\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505508    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-run-ovn\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505633    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-log-socket\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505673    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-kubelet\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505694    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505680    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-socket-dir-parent\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505715    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/228d79c5-beb7-4a41-b877-c68fae3efc15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505718    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-var-lib-kubelet\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505728    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/228d79c5-beb7-4a41-b877-c68fae3efc15-cni-binary-copy\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505749    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-run-k8s-cni-cncf-io\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505779    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/228d79c5-beb7-4a41-b877-c68fae3efc15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505789    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-run-netns\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.506526 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505810    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-cni-dir\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505858    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-var-lib-cni-bin\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505884    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1af79fd6-6d59-4d91-bbee-226e76b02e32-host\") pod \"node-ca-8mvc8\" (UID: \"1af79fd6-6d59-4d91-bbee-226e76b02e32\") " pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505890    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505908    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-var-lib-cni-multus\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505933    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-daemon-config\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505959    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-systemd-units\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.505983    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-slash\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506004    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-var-lib-openvswitch\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506031    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz67b\" (UniqueName: \"kubernetes.io/projected/aa5fd544-ef17-4ad6-8a7d-d68afb85827a-kube-api-access-gz67b\") pod \"iptables-alerter-6bb78\" (UID: \"aa5fd544-ef17-4ad6-8a7d-d68afb85827a\") " pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506032    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6caa7aaf-fa9b-482d-b166-30b9f3515174-tmp-dir\") pod \"node-resolver-dxrln\" (UID: \"6caa7aaf-fa9b-482d-b166-30b9f3515174\") " pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506044    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-hostroot\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506055    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-node-log\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506091    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-cni-bin\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506099    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-cni-dir\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506118    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-cni-netd\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506142    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-var-lib-cni-bin\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qmh\" (UniqueName: \"kubernetes.io/projected/fb7d17e7-5508-41b4-83d5-e5130b662b81-kube-api-access-28qmh\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.507546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506177    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506194    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6caa7aaf-fa9b-482d-b166-30b9f3515174-hosts-file\") pod \"node-resolver-dxrln\" (UID: \"6caa7aaf-fa9b-482d-b166-30b9f3515174\") " pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506203    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-system-cni-dir\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506234    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-conf-dir\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506263    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-run-openvswitch\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506262    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/228d79c5-beb7-4a41-b877-c68fae3efc15-cni-binary-copy\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506279    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-system-cni-dir\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506285    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-node-log\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506320    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-cni-netd\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506325    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-etc-openvswitch\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506343    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-run-systemd\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506372    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-etc-kubernetes\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506394    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506499    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1af79fd6-6d59-4d91-bbee-226e76b02e32-serviceca\") pod \"node-ca-8mvc8\" (UID: \"1af79fd6-6d59-4d91-bbee-226e76b02e32\") " pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506544    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1cf4c16-7f5e-415b-848a-fef07ac8160b-ovnkube-script-lib\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506578    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-cnibin\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506581    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb7d17e7-5508-41b4-83d5-e5130b662b81-multus-daemon-config\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506619    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-cni-bin\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.508489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506667    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb7d17e7-5508-41b4-83d5-e5130b662b81-host-var-lib-cni-multus\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506755    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/228d79c5-beb7-4a41-b877-c68fae3efc15-system-cni-dir\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506047    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1af79fd6-6d59-4d91-bbee-226e76b02e32-host\") pod \"node-ca-8mvc8\" (UID: \"1af79fd6-6d59-4d91-bbee-226e76b02e32\") " pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506792    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/228d79c5-beb7-4a41-b877-c68fae3efc15-cnibin\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506823    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-systemd-units\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506974    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb7d17e7-5508-41b4-83d5-e5130b662b81-cni-binary-copy\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.506994    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-host-slash\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.507034    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1cf4c16-7f5e-415b-848a-fef07ac8160b-var-lib-openvswitch\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.507066    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/228d79c5-beb7-4a41-b877-c68fae3efc15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.507232    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1cf4c16-7f5e-415b-848a-fef07ac8160b-env-overrides\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.507374    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1cf4c16-7f5e-415b-848a-fef07ac8160b-ovnkube-config\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.507374    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa5fd544-ef17-4ad6-8a7d-d68afb85827a-iptables-alerter-script\") pod \"iptables-alerter-6bb78\" (UID: \"aa5fd544-ef17-4ad6-8a7d-d68afb85827a\") " pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.509427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.507467    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/228d79c5-beb7-4a41-b877-c68fae3efc15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.510182 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.510162    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1cf4c16-7f5e-415b-848a-fef07ac8160b-ovn-node-metrics-cert\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.525286 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.525225    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9prg\" (UniqueName: \"kubernetes.io/projected/6caa7aaf-fa9b-482d-b166-30b9f3515174-kube-api-access-g9prg\") pod \"node-resolver-dxrln\" (UID: \"6caa7aaf-fa9b-482d-b166-30b9f3515174\") " pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.525400 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.525333    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz67b\" (UniqueName: \"kubernetes.io/projected/aa5fd544-ef17-4ad6-8a7d-d68afb85827a-kube-api-access-gz67b\") pod \"iptables-alerter-6bb78\" (UID: \"aa5fd544-ef17-4ad6-8a7d-d68afb85827a\") " pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.525778 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.525696    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx77m\" (UniqueName: \"kubernetes.io/projected/1af79fd6-6d59-4d91-bbee-226e76b02e32-kube-api-access-bx77m\") pod \"node-ca-8mvc8\" (UID: \"1af79fd6-6d59-4d91-bbee-226e76b02e32\") " pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.527082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.527044    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nvqs\" (UniqueName: \"kubernetes.io/projected/f1cf4c16-7f5e-415b-848a-fef07ac8160b-kube-api-access-8nvqs\") pod \"ovnkube-node-pjczz\" (UID: \"f1cf4c16-7f5e-415b-848a-fef07ac8160b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.527524 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.527503    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qmh\" (UniqueName: \"kubernetes.io/projected/fb7d17e7-5508-41b4-83d5-e5130b662b81-kube-api-access-28qmh\") pod \"multus-5lkxr\" (UID: \"fb7d17e7-5508-41b4-83d5-e5130b662b81\") " pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.531556 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.531534    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-588k9\" (UniqueName: \"kubernetes.io/projected/228d79c5-beb7-4a41-b877-c68fae3efc15-kube-api-access-588k9\") pod \"multus-additional-cni-plugins-2bgpb\" (UID: \"228d79c5-beb7-4a41-b877-c68fae3efc15\") " pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.589532 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.589278    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42"
Sep 17 18:39:43.601348 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.601166    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:39:43.601449 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:43.601412    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb890a7_3e1f_43fb_96c0_129d39c4f9b0.slice/crio-e0766685e290edb07b5f888ec5cf0c64ec73fd76fe46b82eb2f0348ec25c4b34 WatchSource:0}: Error finding container e0766685e290edb07b5f888ec5cf0c64ec73fd76fe46b82eb2f0348ec25c4b34: Status 404 returned error can't find the container with id e0766685e290edb07b5f888ec5cf0c64ec73fd76fe46b82eb2f0348ec25c4b34
Sep 17 18:39:43.614183 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.614135    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jvtzp"
Sep 17 18:39:43.615502 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.615459    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8mvc8"
Sep 17 18:39:43.615950 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:43.615923    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1463c73f_f522_4ea3_86f1_4ce25079b61c.slice/crio-cd16c2b6cbdfff2dd27ce724877d0db690e29d5596a3e425966e825e52038780 WatchSource:0}: Error finding container cd16c2b6cbdfff2dd27ce724877d0db690e29d5596a3e425966e825e52038780: Status 404 returned error can't find the container with id cd16c2b6cbdfff2dd27ce724877d0db690e29d5596a3e425966e825e52038780
Sep 17 18:39:43.622579 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.622554    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dxrln"
Sep 17 18:39:43.627810 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:43.627739    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b06cb0_be28_4d6f_9bef_694ed58104c8.slice/crio-a19cfae092d50c162ee319b332960a297456cd62a507398d291074d7d6469e50 WatchSource:0}: Error finding container a19cfae092d50c162ee319b332960a297456cd62a507398d291074d7d6469e50: Status 404 returned error can't find the container with id a19cfae092d50c162ee319b332960a297456cd62a507398d291074d7d6469e50
Sep 17 18:39:43.632996 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.632975    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5lkxr"
Sep 17 18:39:43.636685 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.636539    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2bgpb"
Sep 17 18:39:43.638076 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:43.638049    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6caa7aaf_fa9b_482d_b166_30b9f3515174.slice/crio-5e120fdda02b2a248205519e9e06be6ef0dd4addaf1484f2fcb07263f75d5f75 WatchSource:0}: Error finding container 5e120fdda02b2a248205519e9e06be6ef0dd4addaf1484f2fcb07263f75d5f75: Status 404 returned error can't find the container with id 5e120fdda02b2a248205519e9e06be6ef0dd4addaf1484f2fcb07263f75d5f75
Sep 17 18:39:43.646405 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:43.646372    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7d17e7_5508_41b4_83d5_e5130b662b81.slice/crio-a6f2d4a3b5c5e27b62b6b822cdfb09509bb7c8b240354409089a5dd531aee546 WatchSource:0}: Error finding container a6f2d4a3b5c5e27b62b6b822cdfb09509bb7c8b240354409089a5dd531aee546: Status 404 returned error can't find the container with id a6f2d4a3b5c5e27b62b6b822cdfb09509bb7c8b240354409089a5dd531aee546
Sep 17 18:39:43.648121 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.648102    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6bb78"
Sep 17 18:39:43.652011 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:43.651920    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228d79c5_beb7_4a41_b877_c68fae3efc15.slice/crio-5f9fe672de609ca0db1a27f82beb8577d81b923a8305548f5f4a0c34c095a59b WatchSource:0}: Error finding container 5f9fe672de609ca0db1a27f82beb8577d81b923a8305548f5f4a0c34c095a59b: Status 404 returned error can't find the container with id 5f9fe672de609ca0db1a27f82beb8577d81b923a8305548f5f4a0c34c095a59b
Sep 17 18:39:43.654557 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.654364    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:39:43.658776 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:43.658736    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5fd544_ef17_4ad6_8a7d_d68afb85827a.slice/crio-076ee0e63cfb77b671cd45bbdaead0bf28e928aaa836ff63dfaab8dd8e0826be WatchSource:0}: Error finding container 076ee0e63cfb77b671cd45bbdaead0bf28e928aaa836ff63dfaab8dd8e0826be: Status 404 returned error can't find the container with id 076ee0e63cfb77b671cd45bbdaead0bf28e928aaa836ff63dfaab8dd8e0826be
Sep 17 18:39:43.665651 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:39:43.665626    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cf4c16_7f5e_415b_848a_fef07ac8160b.slice/crio-b7123bb1ee51fbea6910513dc9102a2f62cd6496a6b12ea6a9ef2b0836635c3e WatchSource:0}: Error finding container b7123bb1ee51fbea6910513dc9102a2f62cd6496a6b12ea6a9ef2b0836635c3e: Status 404 returned error can't find the container with id b7123bb1ee51fbea6910513dc9102a2f62cd6496a6b12ea6a9ef2b0836635c3e
Sep 17 18:39:43.909249 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.909160    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:43.909249 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:43.909210    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:43.909457 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.909340    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:43.909457 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.909341    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:43.909457 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.909403    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret podName:670d8bfc-6813-421f-941d-c7e35a57f000 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:44.909383651 +0000 UTC m=+5.769208279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret") pod "global-pull-secret-syncer-xnrlh" (UID: "670d8bfc-6813-421f-941d-c7e35a57f000") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:43.909457 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:43.909418    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs podName:e8969506-84fa-4bcc-a5f6-383cbe2b041f nodeName:}" failed. No retries permitted until 2026-09-17 18:39:44.909411798 +0000 UTC m=+5.769236424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs") pod "network-metrics-daemon-vhljw" (UID: "e8969506-84fa-4bcc-a5f6-383cbe2b041f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:44.009574 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.009539    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9vw\" (UniqueName: \"kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw\") pod \"network-check-target-cc2x9\" (UID: \"19bca968-8c00-452e-93fb-8edd90e3f271\") " pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:44.009735 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:44.009697    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:44.009735 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:44.009721    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:44.009735 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:44.009734    2586 projected.go:196] Error preparing data for projected volume kube-api-access-kd9vw for pod openshift-network-diagnostics/network-check-target-cc2x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:44.009904 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:44.009792    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw podName:19bca968-8c00-452e-93fb-8edd90e3f271 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:45.00977591 +0000 UTC m=+5.869600535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kd9vw" (UniqueName: "kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw") pod "network-check-target-cc2x9" (UID: "19bca968-8c00-452e-93fb-8edd90e3f271") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:44.308436 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.308356    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:44.337014 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.336901    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:34:43 +0000 UTC" deadline="2028-03-13 17:25:16.679221332 +0000 UTC"
Sep 17 18:39:44.337014 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.336956    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13030h45m32.342269058s"
Sep 17 18:39:44.453308 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.452737    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:44.453308 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:44.452926    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:39:44.462899 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.462860    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" event={"ID":"f1cf4c16-7f5e-415b-848a-fef07ac8160b","Type":"ContainerStarted","Data":"b7123bb1ee51fbea6910513dc9102a2f62cd6496a6b12ea6a9ef2b0836635c3e"}
Sep 17 18:39:44.464816 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.464680    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6bb78" event={"ID":"aa5fd544-ef17-4ad6-8a7d-d68afb85827a","Type":"ContainerStarted","Data":"076ee0e63cfb77b671cd45bbdaead0bf28e928aaa836ff63dfaab8dd8e0826be"}
Sep 17 18:39:44.466057 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.466025    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2bgpb" event={"ID":"228d79c5-beb7-4a41-b877-c68fae3efc15","Type":"ContainerStarted","Data":"5f9fe672de609ca0db1a27f82beb8577d81b923a8305548f5f4a0c34c095a59b"}
Sep 17 18:39:44.467410 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.467379    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5lkxr" event={"ID":"fb7d17e7-5508-41b4-83d5-e5130b662b81","Type":"ContainerStarted","Data":"a6f2d4a3b5c5e27b62b6b822cdfb09509bb7c8b240354409089a5dd531aee546"}
Sep 17 18:39:44.469040 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.468988    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dxrln" event={"ID":"6caa7aaf-fa9b-482d-b166-30b9f3515174","Type":"ContainerStarted","Data":"5e120fdda02b2a248205519e9e06be6ef0dd4addaf1484f2fcb07263f75d5f75"}
Sep 17 18:39:44.470520 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.470488    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8mvc8" event={"ID":"1af79fd6-6d59-4d91-bbee-226e76b02e32","Type":"ContainerStarted","Data":"f015daf67f7ec6064ef854536c07fc6e4095aad0c6dafa74258e06e015389583"}
Sep 17 18:39:44.473232 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.473207    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xwwg7" event={"ID":"1463c73f-f522-4ea3-86f1-4ce25079b61c","Type":"ContainerStarted","Data":"cd16c2b6cbdfff2dd27ce724877d0db690e29d5596a3e425966e825e52038780"}
Sep 17 18:39:44.474659 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.474633    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jvtzp" event={"ID":"49b06cb0-be28-4d6f-9bef-694ed58104c8","Type":"ContainerStarted","Data":"a19cfae092d50c162ee319b332960a297456cd62a507398d291074d7d6469e50"}
Sep 17 18:39:44.476459 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.476437    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42" event={"ID":"beb890a7-3e1f-43fb-96c0-129d39c4f9b0","Type":"ContainerStarted","Data":"e0766685e290edb07b5f888ec5cf0c64ec73fd76fe46b82eb2f0348ec25c4b34"}
Sep 17 18:39:44.917044 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.916997    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:44.917243 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:44.917076    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:44.917243 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:44.917206    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:44.917415 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:44.917273    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret podName:670d8bfc-6813-421f-941d-c7e35a57f000 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:46.917255041 +0000 UTC m=+7.777079667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret") pod "global-pull-secret-syncer-xnrlh" (UID: "670d8bfc-6813-421f-941d-c7e35a57f000") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:44.917677 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:44.917658    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:44.917754 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:44.917715    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs podName:e8969506-84fa-4bcc-a5f6-383cbe2b041f nodeName:}" failed. No retries permitted until 2026-09-17 18:39:46.917699187 +0000 UTC m=+7.777523818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs") pod "network-metrics-daemon-vhljw" (UID: "e8969506-84fa-4bcc-a5f6-383cbe2b041f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:45.018121 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:45.018085    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9vw\" (UniqueName: \"kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw\") pod \"network-check-target-cc2x9\" (UID: \"19bca968-8c00-452e-93fb-8edd90e3f271\") " pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:45.018343 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:45.018234    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:45.018343 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:45.018250    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:45.018343 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:45.018259    2586 projected.go:196] Error preparing data for projected volume kube-api-access-kd9vw for pod openshift-network-diagnostics/network-check-target-cc2x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:45.018343 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:45.018313    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw podName:19bca968-8c00-452e-93fb-8edd90e3f271 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:47.018295233 +0000 UTC m=+7.878119879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kd9vw" (UniqueName: "kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw") pod "network-check-target-cc2x9" (UID: "19bca968-8c00-452e-93fb-8edd90e3f271") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:45.338158 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:45.338100    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:34:43 +0000 UTC" deadline="2028-04-04 11:16:11.835638419 +0000 UTC"
Sep 17 18:39:45.338158 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:45.338144    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13552h36m26.497499235s"
Sep 17 18:39:45.454998 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:45.454921    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:45.455164 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:45.455051    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:39:45.455453 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:45.455428    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:45.455542 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:45.455526    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:39:46.453014 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:46.452979    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:46.453485 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:46.453198    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:39:46.489447 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:46.489400    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" event={"ID":"cc730dc409eabf98dcfae53f4dbd938e","Type":"ContainerStarted","Data":"3cda1283483a651afd8e50f342ec70072b733e1bfc9f7c315bbcd9bdfa52a01d"}
Sep 17 18:39:46.936336 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:46.935672    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:46.936336 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:46.935730    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:46.936336 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:46.935866    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:46.936336 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:46.935911    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs podName:e8969506-84fa-4bcc-a5f6-383cbe2b041f nodeName:}" failed. No retries permitted until 2026-09-17 18:39:50.935898269 +0000 UTC m=+11.795722895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs") pod "network-metrics-daemon-vhljw" (UID: "e8969506-84fa-4bcc-a5f6-383cbe2b041f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:46.936336 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:46.936222    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:46.936336 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:46.936267    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret podName:670d8bfc-6813-421f-941d-c7e35a57f000 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:50.936252068 +0000 UTC m=+11.796076697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret") pod "global-pull-secret-syncer-xnrlh" (UID: "670d8bfc-6813-421f-941d-c7e35a57f000") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:47.036865 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:47.036125    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9vw\" (UniqueName: \"kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw\") pod \"network-check-target-cc2x9\" (UID: \"19bca968-8c00-452e-93fb-8edd90e3f271\") " pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:47.036865 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:47.036306    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:47.036865 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:47.036325    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:47.036865 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:47.036339    2586 projected.go:196] Error preparing data for projected volume kube-api-access-kd9vw for pod openshift-network-diagnostics/network-check-target-cc2x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:47.036865 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:47.036394    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw podName:19bca968-8c00-452e-93fb-8edd90e3f271 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:51.036377812 +0000 UTC m=+11.896202444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kd9vw" (UniqueName: "kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw") pod "network-check-target-cc2x9" (UID: "19bca968-8c00-452e-93fb-8edd90e3f271") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:47.455162 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:47.455126    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:47.455613 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:47.455271    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:39:47.455689 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:47.455641    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:47.455730 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:47.455706    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:39:48.452357 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:48.452323    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:48.452558 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:48.452487    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:39:49.455218 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:49.454977    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:49.455218 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:49.455114    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:39:49.455218 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:49.455136    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:49.455218 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:49.455213    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:39:50.453169 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:50.453080    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:50.453306 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:50.453206    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:39:50.503863 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:50.503213    2586 generic.go:356] "Generic (PLEG): container finished" podID="cc730dc409eabf98dcfae53f4dbd938e" containerID="3cda1283483a651afd8e50f342ec70072b733e1bfc9f7c315bbcd9bdfa52a01d" exitCode=0
Sep 17 18:39:50.503863 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:50.503265    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" event={"ID":"cc730dc409eabf98dcfae53f4dbd938e","Type":"ContainerDied","Data":"3cda1283483a651afd8e50f342ec70072b733e1bfc9f7c315bbcd9bdfa52a01d"}
Sep 17 18:39:50.970219 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:50.970169    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:50.970389 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:50.970237    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:50.970445 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:50.970396    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:50.970491 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:50.970465    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret podName:670d8bfc-6813-421f-941d-c7e35a57f000 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:58.970444431 +0000 UTC m=+19.830269072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret") pod "global-pull-secret-syncer-xnrlh" (UID: "670d8bfc-6813-421f-941d-c7e35a57f000") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:50.970991 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:50.970934    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:50.971077 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:50.971053    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs podName:e8969506-84fa-4bcc-a5f6-383cbe2b041f nodeName:}" failed. No retries permitted until 2026-09-17 18:39:58.971036677 +0000 UTC m=+19.830861325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs") pod "network-metrics-daemon-vhljw" (UID: "e8969506-84fa-4bcc-a5f6-383cbe2b041f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:51.079381 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:51.079189    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9vw\" (UniqueName: \"kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw\") pod \"network-check-target-cc2x9\" (UID: \"19bca968-8c00-452e-93fb-8edd90e3f271\") " pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:51.079623 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:51.079360    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:51.079683 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:51.079657    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:51.079683 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:51.079680    2586 projected.go:196] Error preparing data for projected volume kube-api-access-kd9vw for pod openshift-network-diagnostics/network-check-target-cc2x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:51.079773 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:51.079751    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw podName:19bca968-8c00-452e-93fb-8edd90e3f271 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:59.079731574 +0000 UTC m=+19.939556205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kd9vw" (UniqueName: "kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw") pod "network-check-target-cc2x9" (UID: "19bca968-8c00-452e-93fb-8edd90e3f271") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:51.454110 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:51.454031    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:51.454270 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:51.454148    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:39:51.454493 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:51.454474    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:51.454584 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:51.454566    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:39:52.453109 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:52.453074    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:52.453498 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:52.453225    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:39:53.452654 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:53.452613    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:53.452816 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:53.452756    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:39:53.453253 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:53.453222    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:53.453626 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:53.453345    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:39:54.452800 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:54.452716    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:54.453002 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:54.452876    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:39:55.455314 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:55.455278    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:55.455776 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:55.455412    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:39:55.455822 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:55.455797    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:55.455918 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:55.455895    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:39:56.453153 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:56.453117    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:56.453360 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:56.453258    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:39:57.452149 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:57.452120    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:57.452537 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:57.452222    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:39:57.452537 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:57.452288    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:57.452537 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:57.452364    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:39:58.452812 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:58.452773    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:58.453320 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:58.452956    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:39:59.036872 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:59.036613    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:59.037064 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:59.036931    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:39:59.037064 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:59.036765    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:59.037064 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:59.037024    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret podName:670d8bfc-6813-421f-941d-c7e35a57f000 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:15.037001238 +0000 UTC m=+35.896825881 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret") pod "global-pull-secret-syncer-xnrlh" (UID: "670d8bfc-6813-421f-941d-c7e35a57f000") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:59.037233 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:59.037086    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:59.037233 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:59.037144    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs podName:e8969506-84fa-4bcc-a5f6-383cbe2b041f nodeName:}" failed. No retries permitted until 2026-09-17 18:40:15.037127565 +0000 UTC m=+35.896952205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs") pod "network-metrics-daemon-vhljw" (UID: "e8969506-84fa-4bcc-a5f6-383cbe2b041f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:59.137328 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:59.137294    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9vw\" (UniqueName: \"kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw\") pod \"network-check-target-cc2x9\" (UID: \"19bca968-8c00-452e-93fb-8edd90e3f271\") " pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:59.137526 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:59.137506    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:59.137588 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:59.137532    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:59.137588 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:59.137545    2586 projected.go:196] Error preparing data for projected volume kube-api-access-kd9vw for pod openshift-network-diagnostics/network-check-target-cc2x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:59.137686 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:59.137617    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw podName:19bca968-8c00-452e-93fb-8edd90e3f271 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:15.137597365 +0000 UTC m=+35.997422016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kd9vw" (UniqueName: "kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw") pod "network-check-target-cc2x9" (UID: "19bca968-8c00-452e-93fb-8edd90e3f271") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:59.452229 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:59.452143    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:39:59.452401 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:59.452268    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:39:59.452401 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:39:59.452303    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:39:59.452485 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:39:59.452406    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:40:00.452334 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:00.452300    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:00.452731 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:00.452413    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:40:01.453323 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:01.453253    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:01.453668 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:01.453354    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:40:01.453668 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:01.453432    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:01.453668 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:01.453533    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:40:01.542784 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:01.542699    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" event={"ID":"894af7d1d57be17aaf3ffac63d929c3b","Type":"ContainerStarted","Data":"57f8845767896029316e7360bd4da0d12f2ecb9ea6537ff34bd45c82cfffe643"}
Sep 17 18:40:02.453491 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.453243    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:02.454191 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:02.453617    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:40:02.547165 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.547136    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" event={"ID":"cc730dc409eabf98dcfae53f4dbd938e","Type":"ContainerStarted","Data":"9bfdc903746190e1ed4a002a956128c29ea830076eaa7715010444bc777d7b46"}
Sep 17 18:40:02.548702 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.548678    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jvtzp" event={"ID":"49b06cb0-be28-4d6f-9bef-694ed58104c8","Type":"ContainerStarted","Data":"0c407a586ce9ae8db2389f787c6354e82de3412c3117409361b6a430035229e3"}
Sep 17 18:40:02.550115 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.550096    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42" event={"ID":"beb890a7-3e1f-43fb-96c0-129d39c4f9b0","Type":"ContainerStarted","Data":"cf5d034ba232f1c7029f634ffb68be68d6d74d508dc2abfa5506f7156fb1de8d"}
Sep 17 18:40:02.552878 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.552863    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:40:02.553208 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.553188    2586 generic.go:356] "Generic (PLEG): container finished" podID="f1cf4c16-7f5e-415b-848a-fef07ac8160b" containerID="0ec8e13234737838a513edaca919f41a88544a6908eac47e5489fcd5a45c46bd" exitCode=1
Sep 17 18:40:02.553289 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.553258    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" event={"ID":"f1cf4c16-7f5e-415b-848a-fef07ac8160b","Type":"ContainerStarted","Data":"33bf8af3959076f9fdd5304d4d073b23dec8980b8f0db01bb665b3f7ce348949"}
Sep 17 18:40:02.553329 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.553307    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" event={"ID":"f1cf4c16-7f5e-415b-848a-fef07ac8160b","Type":"ContainerStarted","Data":"b361132eeeb449e4be1a4ccfc3bcdf725dd831e1600e09c1c99efae8d5c81afb"}
Sep 17 18:40:02.553368 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.553345    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" event={"ID":"f1cf4c16-7f5e-415b-848a-fef07ac8160b","Type":"ContainerStarted","Data":"9f2e1f3d9cc430961279f9f856a52289d5fc0ff1dcac101b41b05f753a0254ea"}
Sep 17 18:40:02.553406 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.553379    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" event={"ID":"f1cf4c16-7f5e-415b-848a-fef07ac8160b","Type":"ContainerStarted","Data":"87ac6caf49e485ad29ec9f6caf5ad63fee15a6494117efa72d36c1f50500c783"}
Sep 17 18:40:02.553453 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.553409    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" event={"ID":"f1cf4c16-7f5e-415b-848a-fef07ac8160b","Type":"ContainerDied","Data":"0ec8e13234737838a513edaca919f41a88544a6908eac47e5489fcd5a45c46bd"}
Sep 17 18:40:02.553453 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.553440    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" event={"ID":"f1cf4c16-7f5e-415b-848a-fef07ac8160b","Type":"ContainerStarted","Data":"55599fa0c977224dad109bd4c8a598135a9f7b90877b165fb6e2de78032fce42"}
Sep 17 18:40:02.554809 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.554787    2586 generic.go:356] "Generic (PLEG): container finished" podID="228d79c5-beb7-4a41-b877-c68fae3efc15" containerID="ca152c76e900d7f6e0b500e67323226f6a74730f5ee13c2cc96fd1dc4e0836aa" exitCode=0
Sep 17 18:40:02.554910 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.554870    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2bgpb" event={"ID":"228d79c5-beb7-4a41-b877-c68fae3efc15","Type":"ContainerDied","Data":"ca152c76e900d7f6e0b500e67323226f6a74730f5ee13c2cc96fd1dc4e0836aa"}
Sep 17 18:40:02.556436 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.556415    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5lkxr" event={"ID":"fb7d17e7-5508-41b4-83d5-e5130b662b81","Type":"ContainerStarted","Data":"176655a313c0ae1bb3069cbf9b1ab2e195f9459dda6a9f33ca9ca285d70d5835"}
Sep 17 18:40:02.557948 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.557917    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dxrln" event={"ID":"6caa7aaf-fa9b-482d-b166-30b9f3515174","Type":"ContainerStarted","Data":"8eb28048de15611499cc2c2a048a43c489b6a9f05938f582e9e8a5b362b2cd75"}
Sep 17 18:40:02.559329 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.559294    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8mvc8" event={"ID":"1af79fd6-6d59-4d91-bbee-226e76b02e32","Type":"ContainerStarted","Data":"7ef98d5ab8478faf767087afec6b45dd648db44180bee3a550929a45f7559a9a"}
Sep 17 18:40:02.560480 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.560461    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xwwg7" event={"ID":"1463c73f-f522-4ea3-86f1-4ce25079b61c","Type":"ContainerStarted","Data":"24b87ce01980ef7f56b567714f2f0ddd03fbcd2269beabb632eb9e870f422d14"}
Sep 17 18:40:02.566572 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.566518    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-204.ec2.internal" podStartSLOduration=20.566505357 podStartE2EDuration="20.566505357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:01.602306501 +0000 UTC m=+22.462131163" watchObservedRunningTime="2026-09-17 18:40:02.566505357 +0000 UTC m=+23.426330005"
Sep 17 18:40:02.584004 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.583963    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-204.ec2.internal" podStartSLOduration=20.583948736 podStartE2EDuration="20.583948736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:02.56721214 +0000 UTC m=+23.427036814" watchObservedRunningTime="2026-09-17 18:40:02.583948736 +0000 UTC m=+23.443773405"
Sep 17 18:40:02.584476 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.584454    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xwwg7" podStartSLOduration=7.454160147 podStartE2EDuration="21.584446052s" totalImagesPullingTime="14.130285905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:02.58373956 +0000 UTC m=+23.443564221" watchObservedRunningTime="2026-09-17 18:40:02.584446052 +0000 UTC m=+23.444270772"
Sep 17 18:40:02.622299 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.622240    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8mvc8" podStartSLOduration=4.077658618 podStartE2EDuration="21.622218157s" totalImagesPullingTime="17.544559539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:02.603321791 +0000 UTC m=+23.463146463" watchObservedRunningTime="2026-09-17 18:40:02.622218157 +0000 UTC m=+23.482042806"
Sep 17 18:40:02.622805 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.622775    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jvtzp" podStartSLOduration=3.751568878 podStartE2EDuration="21.622767639s" totalImagesPullingTime="17.871198761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:02.622049339 +0000 UTC m=+23.481873997" watchObservedRunningTime="2026-09-17 18:40:02.622767639 +0000 UTC m=+23.482592289"
Sep 17 18:40:02.636826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.636776    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dxrln" podStartSLOduration=4.097165957 podStartE2EDuration="21.636760156s" totalImagesPullingTime="17.539594199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:02.636087236 +0000 UTC m=+23.495911905" watchObservedRunningTime="2026-09-17 18:40:02.636760156 +0000 UTC m=+23.496584803"
Sep 17 18:40:02.654854 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:02.654788    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5lkxr" podStartSLOduration=3.734779282 podStartE2EDuration="21.654772553s" totalImagesPullingTime="17.919993271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:02.654274553 +0000 UTC m=+23.514099220" watchObservedRunningTime="2026-09-17 18:40:02.654772553 +0000 UTC m=+23.514597201"
Sep 17 18:40:03.273025 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:03.272735    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:40:03.389655 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:03.389442    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:40:03.273020828Z","UUID":"1495d920-a96d-4ef8-93c7-8a6271fdef11","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:40:03.390861 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:03.390825    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:40:03.390976 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:03.390876    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:40:03.452342 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:03.452314    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:03.452507 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:03.452314    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:03.452507 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:03.452428    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:40:03.452632 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:03.452532    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:40:03.564389 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:03.564352    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6bb78" event={"ID":"aa5fd544-ef17-4ad6-8a7d-d68afb85827a","Type":"ContainerStarted","Data":"40551da5768dfda6e07bd446cd7f96bbebc3191354fc6ff63be1dc0a8b7f8df9"}
Sep 17 18:40:03.566556 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:03.566530    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42" event={"ID":"beb890a7-3e1f-43fb-96c0-129d39c4f9b0","Type":"ContainerStarted","Data":"b7fb0dd5bbbeb235568b08f040482d4b1c9ae9a13b16d70ba5c4b35b65768583"}
Sep 17 18:40:03.590740 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:03.590676    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6bb78" podStartSLOduration=4.777703763 podStartE2EDuration="22.5906548s" totalImagesPullingTime="17.812951037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:03.590544843 +0000 UTC m=+24.450369509" watchObservedRunningTime="2026-09-17 18:40:03.5906548 +0000 UTC m=+24.450479446"
Sep 17 18:40:04.452890 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:04.452866    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:04.453013 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:04.452988    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:40:04.573304 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:04.573273    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:40:04.574392 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:04.573930    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" event={"ID":"f1cf4c16-7f5e-415b-848a-fef07ac8160b","Type":"ContainerStarted","Data":"fe8ac80654a35c3f793a9bbb4487ec1f75684e65d9e4369869fd7d57c663b8d8"}
Sep 17 18:40:05.453038 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:05.453003    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:05.453212 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:05.453009    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:05.453212 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:05.453148    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:40:05.453212 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:05.453197    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:40:05.578991 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:05.578952    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42" event={"ID":"beb890a7-3e1f-43fb-96c0-129d39c4f9b0","Type":"ContainerStarted","Data":"f9b34dcfed6be98bd8a598723e240e83a6e1c2a9a19bfab094cf7afa18581e3b"}
Sep 17 18:40:05.603905 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:05.603820    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8w42" podStartSLOduration=4.072569061 podStartE2EDuration="24.603796927s" totalImagesPullingTime="20.531227866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:05.603376914 +0000 UTC m=+26.463201552" watchObservedRunningTime="2026-09-17 18:40:05.603796927 +0000 UTC m=+26.463621580"
Sep 17 18:40:06.452989 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:06.452956    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:06.453168 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:06.453081    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:40:07.452710 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.452519    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:07.453157 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.452550    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:07.453157 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:07.452813    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:40:07.453157 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:07.452886    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:40:07.525900 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.525874    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:40:07.526449 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.526432    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:40:07.586749 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.586724    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:40:07.587128 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.587108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" event={"ID":"f1cf4c16-7f5e-415b-848a-fef07ac8160b","Type":"ContainerStarted","Data":"b08b34680166d2b947595a44ff3af8c22870bf6828ed63edabfbc666dfba0ed2"}
Sep 17 18:40:07.587354 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.587339    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:40:07.587570 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.587538    2586 scope.go:117] "RemoveContainer" containerID="0ec8e13234737838a513edaca919f41a88544a6908eac47e5489fcd5a45c46bd"
Sep 17 18:40:07.589192 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.589156    2586 generic.go:356] "Generic (PLEG): container finished" podID="228d79c5-beb7-4a41-b877-c68fae3efc15" containerID="40fecfb2c0b0392aa78568989e94f11da5de1547e95f2a87f23fe69b0518ae16" exitCode=0
Sep 17 18:40:07.589287 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.589213    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2bgpb" event={"ID":"228d79c5-beb7-4a41-b877-c68fae3efc15","Type":"ContainerDied","Data":"40fecfb2c0b0392aa78568989e94f11da5de1547e95f2a87f23fe69b0518ae16"}
Sep 17 18:40:07.589459 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.589445    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:40:07.590122 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.590106    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xwwg7"
Sep 17 18:40:07.604276 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:07.604253    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:40:08.452548 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:08.452505    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:08.452744 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:08.452681    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:40:08.595032 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:08.595001    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:40:08.595442 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:08.595416    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" event={"ID":"f1cf4c16-7f5e-415b-848a-fef07ac8160b","Type":"ContainerStarted","Data":"62358fac8e6cbd72441e5eaee7b8837ff4b6ac0ea547b861b842b75687b61a22"}
Sep 17 18:40:08.595695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:08.595576    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:40:08.595695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:08.595649    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:40:08.611223 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:08.611194    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:40:08.639961 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:08.639908    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz" podStartSLOduration=9.735721371 podStartE2EDuration="27.639888441s" totalImagesPullingTime="17.90416707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:08.639399291 +0000 UTC m=+29.499223959" watchObservedRunningTime="2026-09-17 18:40:08.639888441 +0000 UTC m=+29.499713091"
Sep 17 18:40:09.118263 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:09.118007    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xnrlh"]
Sep 17 18:40:09.118450 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:09.118291    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:09.118450 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:09.118401    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:40:09.125154 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:09.125046    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vhljw"]
Sep 17 18:40:09.125154 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:09.125153    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:09.125354 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:09.125249    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:40:09.125653 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:09.125631    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cc2x9"]
Sep 17 18:40:09.125762 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:09.125728    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:09.125847 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:09.125813    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:40:09.599935 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:09.599901    2586 generic.go:356] "Generic (PLEG): container finished" podID="228d79c5-beb7-4a41-b877-c68fae3efc15" containerID="ff215ab321ea7af021a74fd9e8af43168e68352297094ef95a11ae1df559dc3d" exitCode=0
Sep 17 18:40:09.600409 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:09.599990    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2bgpb" event={"ID":"228d79c5-beb7-4a41-b877-c68fae3efc15","Type":"ContainerDied","Data":"ff215ab321ea7af021a74fd9e8af43168e68352297094ef95a11ae1df559dc3d"}
Sep 17 18:40:10.452764 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:10.452730    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:10.452764 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:10.452772    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:10.452996 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:10.452898    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:40:10.453041 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:10.453011    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:40:11.453100 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:11.453072    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:11.453439 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:11.453153    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:40:11.606873 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:11.606824    2586 generic.go:356] "Generic (PLEG): container finished" podID="228d79c5-beb7-4a41-b877-c68fae3efc15" containerID="f39b41db7fa301ffaead3d8cdb130f1b47f0f4b626699a8fa87b900808100402" exitCode=0
Sep 17 18:40:11.606873 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:11.606866    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2bgpb" event={"ID":"228d79c5-beb7-4a41-b877-c68fae3efc15","Type":"ContainerDied","Data":"f39b41db7fa301ffaead3d8cdb130f1b47f0f4b626699a8fa87b900808100402"}
Sep 17 18:40:12.452129 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:12.452091    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:12.452129 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:12.452131    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:12.452330 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:12.452213    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:40:12.452370 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:12.452330    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:40:12.880146 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:12.880118    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dxrln_6caa7aaf-fa9b-482d-b166-30b9f3515174/dns-node-resolver/0.log"
Sep 17 18:40:13.452347 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:13.452313    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:13.452514 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:13.452422    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnrlh" podUID="670d8bfc-6813-421f-941d-c7e35a57f000"
Sep 17 18:40:14.039162 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:14.039132    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8mvc8_1af79fd6-6d59-4d91-bbee-226e76b02e32/node-ca/0.log"
Sep 17 18:40:14.453166 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:14.453135    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:14.453354 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:14.453175    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:14.453354 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:14.453246    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc2x9" podUID="19bca968-8c00-452e-93fb-8edd90e3f271"
Sep 17 18:40:14.453432 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:14.453364    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vhljw" podUID="e8969506-84fa-4bcc-a5f6-383cbe2b041f"
Sep 17 18:40:14.735805 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:14.735777    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-204.ec2.internal" event="NodeReady"
Sep 17 18:40:14.735975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:14.735917    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:40:15.062277 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:15.062194    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:15.062277 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:15.062258    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:15.062687 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:15.062364    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:40:15.062687 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:15.062363    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:40:15.062687 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:15.062434    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs podName:e8969506-84fa-4bcc-a5f6-383cbe2b041f nodeName:}" failed. No retries permitted until 2026-09-17 18:40:47.062415223 +0000 UTC m=+67.922239848 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs") pod "network-metrics-daemon-vhljw" (UID: "e8969506-84fa-4bcc-a5f6-383cbe2b041f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:40:15.062687 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:15.062455    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret podName:670d8bfc-6813-421f-941d-c7e35a57f000 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:47.062444369 +0000 UTC m=+67.922268995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret") pod "global-pull-secret-syncer-xnrlh" (UID: "670d8bfc-6813-421f-941d-c7e35a57f000") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:40:15.163518 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:15.163481    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9vw\" (UniqueName: \"kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw\") pod \"network-check-target-cc2x9\" (UID: \"19bca968-8c00-452e-93fb-8edd90e3f271\") " pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:15.163748 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:15.163623    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:40:15.163748 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:15.163638    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:40:15.163748 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:15.163647    2586 projected.go:196] Error preparing data for projected volume kube-api-access-kd9vw for pod openshift-network-diagnostics/network-check-target-cc2x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:40:15.163748 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:15.163699    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw podName:19bca968-8c00-452e-93fb-8edd90e3f271 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:47.163684193 +0000 UTC m=+68.023508819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kd9vw" (UniqueName: "kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw") pod "network-check-target-cc2x9" (UID: "19bca968-8c00-452e-93fb-8edd90e3f271") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:40:15.452881 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:15.452848    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:15.456920 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:15.456885    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:40:16.453079 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:16.453039    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:16.453647 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:16.453050    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:16.456039 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:16.456015    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:40:16.456335 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:16.456319    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:40:16.457243 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:16.457067    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5jd5p\""
Sep 17 18:40:16.457243 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:16.457167    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:40:16.457425 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:16.457393    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7pg8w\""
Sep 17 18:40:17.623815 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:17.623491    2586 generic.go:356] "Generic (PLEG): container finished" podID="228d79c5-beb7-4a41-b877-c68fae3efc15" containerID="32447a8728e8dc575474eeccbbad555ad7f664bbcd2b543c35da0674421d77db" exitCode=0
Sep 17 18:40:17.623815 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:17.623592    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2bgpb" event={"ID":"228d79c5-beb7-4a41-b877-c68fae3efc15","Type":"ContainerDied","Data":"32447a8728e8dc575474eeccbbad555ad7f664bbcd2b543c35da0674421d77db"}
Sep 17 18:40:18.632410 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:18.632373    2586 generic.go:356] "Generic (PLEG): container finished" podID="228d79c5-beb7-4a41-b877-c68fae3efc15" containerID="6c6dce0e75632fcfcf15cf811e0ae1866631bfadb18770ea0c31d8e39e9a8d98" exitCode=0
Sep 17 18:40:18.632791 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:18.632416    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2bgpb" event={"ID":"228d79c5-beb7-4a41-b877-c68fae3efc15","Type":"ContainerDied","Data":"6c6dce0e75632fcfcf15cf811e0ae1866631bfadb18770ea0c31d8e39e9a8d98"}
Sep 17 18:40:19.638888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:19.638856    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2bgpb" event={"ID":"228d79c5-beb7-4a41-b877-c68fae3efc15","Type":"ContainerStarted","Data":"cb9c4374e4efcc2ad8d306108543a7f6dd30ba3c46adbc6ce0489f640b7ea9a3"}
Sep 17 18:40:34.805244 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.805180    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2bgpb" podStartSLOduration=23.011678959 podStartE2EDuration="53.805159485s" totalImagesPullingTime="30.793480526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:19.672808481 +0000 UTC m=+40.532633121" watchObservedRunningTime="2026-09-17 18:40:34.805159485 +0000 UTC m=+55.664984134"
Sep 17 18:40:34.806060 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.806036    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fd87b49f5-db6rb"]
Sep 17 18:40:34.858049 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.858013    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fd87b49f5-db6rb"]
Sep 17 18:40:34.858225 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.858120    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd87b49f5-db6rb"
Sep 17 18:40:34.875394 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.875368    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd87b49f5-db6rb"
Sep 17 18:40:34.884657 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.884632    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cx55j"]
Sep 17 18:40:34.895977 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.895949    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-66znr"]
Sep 17 18:40:34.896108 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.896068    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cx55j"
Sep 17 18:40:34.900318 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.900295    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:40:34.900318 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.900314    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:40:34.900457 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.900314    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l4x2p\""
Sep 17 18:40:34.900692 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.900678    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:40:34.910263 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.910244    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:34.913676 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.913658    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:40:34.914102 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.914085    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:40:34.921110 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.921092    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:40:34.921945 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.921928    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kzdh4\""
Sep 17 18:40:34.925992 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.925962    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:40:34.926742 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.926725    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fd87b49f5-db6rb"]
Sep 17 18:40:34.945067 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.945042    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cx55j"]
Sep 17 18:40:34.945668 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:34.945651    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-66znr"]
Sep 17 18:40:35.002808 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.002774    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.003012 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.002827    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.003012 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.002894    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-data-volume\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.003012 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.002937    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zcg\" (UniqueName: \"kubernetes.io/projected/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-kube-api-access-j4zcg\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.003012 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.002979    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-crio-socket\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.003012 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.003013    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5f74\" (UniqueName: \"kubernetes.io/projected/584cdffd-f790-4665-acc8-f7cdd2d1f46f-kube-api-access-w5f74\") pod \"ingress-canary-cx55j\" (UID: \"584cdffd-f790-4665-acc8-f7cdd2d1f46f\") " pod="openshift-ingress-canary/ingress-canary-cx55j"
Sep 17 18:40:35.003198 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.003044    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/584cdffd-f790-4665-acc8-f7cdd2d1f46f-cert\") pod \"ingress-canary-cx55j\" (UID: \"584cdffd-f790-4665-acc8-f7cdd2d1f46f\") " pod="openshift-ingress-canary/ingress-canary-cx55j"
Sep 17 18:40:35.071975 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.071945    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tx5nx"]
Sep 17 18:40:35.095086 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.095058    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.098230 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.098205    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tx5nx"]
Sep 17 18:40:35.098378 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.098293    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gj7h2\""
Sep 17 18:40:35.098378 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.098305    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:40:35.098473 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.098385    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:40:35.103869 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.103824    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/584cdffd-f790-4665-acc8-f7cdd2d1f46f-cert\") pod \"ingress-canary-cx55j\" (UID: \"584cdffd-f790-4665-acc8-f7cdd2d1f46f\") " pod="openshift-ingress-canary/ingress-canary-cx55j"
Sep 17 18:40:35.103981 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.103889    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.103981 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.103935    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.103981 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.103973    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-data-volume\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.104133 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.103995    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zcg\" (UniqueName: \"kubernetes.io/projected/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-kube-api-access-j4zcg\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.104133 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.104030    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-crio-socket\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.104133 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.104111    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-crio-socket\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.104268 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.104145    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5f74\" (UniqueName: \"kubernetes.io/projected/584cdffd-f790-4665-acc8-f7cdd2d1f46f-kube-api-access-w5f74\") pod \"ingress-canary-cx55j\" (UID: \"584cdffd-f790-4665-acc8-f7cdd2d1f46f\") " pod="openshift-ingress-canary/ingress-canary-cx55j"
Sep 17 18:40:35.104323 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.104286    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-data-volume\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.104509 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.104485    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.107852 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.107813    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.107994 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.107978    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/584cdffd-f790-4665-acc8-f7cdd2d1f46f-cert\") pod \"ingress-canary-cx55j\" (UID: \"584cdffd-f790-4665-acc8-f7cdd2d1f46f\") " pod="openshift-ingress-canary/ingress-canary-cx55j"
Sep 17 18:40:35.123539 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.123500    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zcg\" (UniqueName: \"kubernetes.io/projected/81b4a5e2-4e3d-416f-b42b-3e43ceb3365c-kube-api-access-j4zcg\") pod \"insights-runtime-extractor-66znr\" (UID: \"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c\") " pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.125100 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.125078    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5f74\" (UniqueName: \"kubernetes.io/projected/584cdffd-f790-4665-acc8-f7cdd2d1f46f-kube-api-access-w5f74\") pod \"ingress-canary-cx55j\" (UID: \"584cdffd-f790-4665-acc8-f7cdd2d1f46f\") " pod="openshift-ingress-canary/ingress-canary-cx55j"
Sep 17 18:40:35.204660 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.204605    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cx55j"
Sep 17 18:40:35.204873 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.204850    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9bf4517-1532-40ee-b569-0aed944e652b-metrics-tls\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.204940 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.204888    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf4517-1532-40ee-b569-0aed944e652b-config-volume\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.204940 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.204902    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bf4517-1532-40ee-b569-0aed944e652b-tmp-dir\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.204940 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.204916    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttfph\" (UniqueName: \"kubernetes.io/projected/d9bf4517-1532-40ee-b569-0aed944e652b-kube-api-access-ttfph\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.217955 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.217924    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-66znr"
Sep 17 18:40:35.305599 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.305571    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9bf4517-1532-40ee-b569-0aed944e652b-metrics-tls\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.305772 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.305613    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf4517-1532-40ee-b569-0aed944e652b-config-volume\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.305772 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.305629    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bf4517-1532-40ee-b569-0aed944e652b-tmp-dir\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.305772 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.305651    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttfph\" (UniqueName: \"kubernetes.io/projected/d9bf4517-1532-40ee-b569-0aed944e652b-kube-api-access-ttfph\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.306082 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.306059    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bf4517-1532-40ee-b569-0aed944e652b-tmp-dir\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.306221 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.306203    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf4517-1532-40ee-b569-0aed944e652b-config-volume\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.307969 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.307953    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9bf4517-1532-40ee-b569-0aed944e652b-metrics-tls\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.321066 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.321041    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttfph\" (UniqueName: \"kubernetes.io/projected/d9bf4517-1532-40ee-b569-0aed944e652b-kube-api-access-ttfph\") pod \"dns-default-tx5nx\" (UID: \"d9bf4517-1532-40ee-b569-0aed944e652b\") " pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.405136 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.405098    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:35.568460 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.568429    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cx55j"]
Sep 17 18:40:35.571756 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:35.571729    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584cdffd_f790_4665_acc8_f7cdd2d1f46f.slice/crio-77e60faec7f57879816ea583e8bcf1a1d08b5849d43ebccf90b5fe2f57e0fe36 WatchSource:0}: Error finding container 77e60faec7f57879816ea583e8bcf1a1d08b5849d43ebccf90b5fe2f57e0fe36: Status 404 returned error can't find the container with id 77e60faec7f57879816ea583e8bcf1a1d08b5849d43ebccf90b5fe2f57e0fe36
Sep 17 18:40:35.572711 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.572684    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-66znr"]
Sep 17 18:40:35.575565 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:35.575542    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b4a5e2_4e3d_416f_b42b_3e43ceb3365c.slice/crio-af6a7ccfc5b654582faea8932704cd5dea8fad54eb5520ef6bb02a0660db71a2 WatchSource:0}: Error finding container af6a7ccfc5b654582faea8932704cd5dea8fad54eb5520ef6bb02a0660db71a2: Status 404 returned error can't find the container with id af6a7ccfc5b654582faea8932704cd5dea8fad54eb5520ef6bb02a0660db71a2
Sep 17 18:40:35.677647 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.677620    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-66znr" event={"ID":"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c","Type":"ContainerStarted","Data":"af6a7ccfc5b654582faea8932704cd5dea8fad54eb5520ef6bb02a0660db71a2"}
Sep 17 18:40:35.678732 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.678704    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cx55j" event={"ID":"584cdffd-f790-4665-acc8-f7cdd2d1f46f","Type":"ContainerStarted","Data":"77e60faec7f57879816ea583e8bcf1a1d08b5849d43ebccf90b5fe2f57e0fe36"}
Sep 17 18:40:35.678848 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.678758    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd87b49f5-db6rb"
Sep 17 18:40:35.703282 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.703258    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tx5nx"]
Sep 17 18:40:35.706363 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:35.706329    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bf4517_1532_40ee_b569_0aed944e652b.slice/crio-c819835a916380f42ccc5537c11e2e22c15b135a67d88fca9b1db2454714986e WatchSource:0}: Error finding container c819835a916380f42ccc5537c11e2e22c15b135a67d88fca9b1db2454714986e: Status 404 returned error can't find the container with id c819835a916380f42ccc5537c11e2e22c15b135a67d88fca9b1db2454714986e
Sep 17 18:40:35.728450 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.728418    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fd87b49f5-db6rb"]
Sep 17 18:40:35.729582 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:35.729560    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fd87b49f5-db6rb"]
Sep 17 18:40:36.683782 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:36.683737    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-66znr" event={"ID":"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c","Type":"ContainerStarted","Data":"72437fb9f35dd4460c3b6c9c934607ff41269f923791885c046d159cd6a7a16c"}
Sep 17 18:40:36.685261 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:36.685191    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tx5nx" event={"ID":"d9bf4517-1532-40ee-b569-0aed944e652b","Type":"ContainerStarted","Data":"c819835a916380f42ccc5537c11e2e22c15b135a67d88fca9b1db2454714986e"}
Sep 17 18:40:37.689113 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:37.689081    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-66znr" event={"ID":"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c","Type":"ContainerStarted","Data":"da26cc117eadee322e6f5e7dc8a58d5da950eada4bdfd4818eae2c2209b220fb"}
Sep 17 18:40:39.695587 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:39.695562    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cx55j" event={"ID":"584cdffd-f790-4665-acc8-f7cdd2d1f46f","Type":"ContainerStarted","Data":"55dd5da83e4cbc64fe7b222240c10914aec38e180ffc5b76f3f022e252da3cb0"}
Sep 17 18:40:39.697092 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:39.697074    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tx5nx" event={"ID":"d9bf4517-1532-40ee-b569-0aed944e652b","Type":"ContainerStarted","Data":"3447e79561d3377eb3fe474abb452941edd851a7280e1f646159d8a9e2105308"}
Sep 17 18:40:40.612881 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:40.612826    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjczz"
Sep 17 18:40:40.651068 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:40.651008    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cx55j" podStartSLOduration=2.912414532 podStartE2EDuration="6.650991619s" totalImagesPullingTime="3.738577087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:39.715204506 +0000 UTC m=+60.575029171" watchObservedRunningTime="2026-09-17 18:40:40.650991619 +0000 UTC m=+61.510816270"
Sep 17 18:40:40.702522 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:40.702484    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-66znr" event={"ID":"81b4a5e2-4e3d-416f-b42b-3e43ceb3365c","Type":"ContainerStarted","Data":"a9882d4dbc2807448bef50f0f30dab234397e69265c058913c813c83fec5345c"}
Sep 17 18:40:40.704413 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:40.704387    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tx5nx" event={"ID":"d9bf4517-1532-40ee-b569-0aed944e652b","Type":"ContainerStarted","Data":"b79e561e46eb018b554edd8232607be7ebeb184a188b6fcf52fa3810f06f4687"}
Sep 17 18:40:40.704558 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:40.704538    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:40.732684 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:40.732629    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-66znr" podStartSLOduration=2.9899240049999998 podStartE2EDuration="6.732613322s" totalImagesPullingTime="3.742689317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:40:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:40.731564167 +0000 UTC m=+61.591388834" watchObservedRunningTime="2026-09-17 18:40:40.732613322 +0000 UTC m=+61.592437966"
Sep 17 18:40:40.759765 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:40.759704    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tx5nx" podStartSLOduration=2.155419981 podStartE2EDuration="5.759688023s" totalImagesPullingTime="3.604268042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:40.759301015 +0000 UTC m=+61.619125679" watchObservedRunningTime="2026-09-17 18:40:40.759688023 +0000 UTC m=+61.619512670"
Sep 17 18:40:41.878317 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.878280    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-885b84785-k62rw"]
Sep 17 18:40:41.881204 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.881182    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:41.884925 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.884899    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 18:40:41.885075 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.884899    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 18:40:41.885075 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.884932    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 18:40:41.885075 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.884933    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pvqrx\""
Sep 17 18:40:41.885075 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.885056    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 18:40:41.885075 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.884935    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 18:40:41.885075 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.885067    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 18:40:41.885395 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.885010    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 18:40:41.894827 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.894806    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-885b84785-k62rw"]
Sep 17 18:40:41.955212 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.955176    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpwh4\" (UniqueName: \"kubernetes.io/projected/98b10151-b464-420a-95b9-4367a22e5e92-kube-api-access-kpwh4\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:41.955212 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.955209    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-oauth-serving-cert\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:41.955423 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.955242    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-service-ca\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:41.955423 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.955269    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-oauth-config\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:41.955423 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.955292    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-serving-cert\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:41.955423 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:41.955310    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-console-config\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.055592 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.055560    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpwh4\" (UniqueName: \"kubernetes.io/projected/98b10151-b464-420a-95b9-4367a22e5e92-kube-api-access-kpwh4\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.055592 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.055596    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-oauth-serving-cert\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.055847 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.055687    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-service-ca\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.055847 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.055716    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-oauth-config\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.055847 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.055748    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-serving-cert\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.055847 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.055775    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-console-config\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.056509 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.056484    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-oauth-serving-cert\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.056509 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.056504    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-console-config\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.056700 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.056680    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-service-ca\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.058245 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.058220    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-oauth-config\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.058753 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.058734    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-serving-cert\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.066200 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.066176    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpwh4\" (UniqueName: \"kubernetes.io/projected/98b10151-b464-420a-95b9-4367a22e5e92-kube-api-access-kpwh4\") pod \"console-885b84785-k62rw\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") " pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.189988 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.189905    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:42.470265 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.470190    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-885b84785-k62rw"]
Sep 17 18:40:42.473330 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:42.473305    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b10151_b464_420a_95b9_4367a22e5e92.slice/crio-c96a7b4fd19e252664a217024befb2d625c016bc326187a080bffedffda3f757 WatchSource:0}: Error finding container c96a7b4fd19e252664a217024befb2d625c016bc326187a080bffedffda3f757: Status 404 returned error can't find the container with id c96a7b4fd19e252664a217024befb2d625c016bc326187a080bffedffda3f757
Sep 17 18:40:42.712546 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:42.712502    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-885b84785-k62rw" event={"ID":"98b10151-b464-420a-95b9-4367a22e5e92","Type":"ContainerStarted","Data":"c96a7b4fd19e252664a217024befb2d625c016bc326187a080bffedffda3f757"}
Sep 17 18:40:46.387197 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:46.387164    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"]
Sep 17 18:40:46.389323 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:46.389305    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"
Sep 17 18:40:46.391787 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:46.391762    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lv4rp\""
Sep 17 18:40:46.391937 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:46.391852    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 18:40:46.400581 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:46.400557    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"]
Sep 17 18:40:46.486402 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:46.486362    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d3997c3c-be8b-49dc-aeda-8e39adfd65e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-59x8f\" (UID: \"d3997c3c-be8b-49dc-aeda-8e39adfd65e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"
Sep 17 18:40:46.586710 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:46.586677    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d3997c3c-be8b-49dc-aeda-8e39adfd65e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-59x8f\" (UID: \"d3997c3c-be8b-49dc-aeda-8e39adfd65e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"
Sep 17 18:40:46.586866 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:46.586824    2586 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 18:40:46.586919 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:46.586909    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3997c3c-be8b-49dc-aeda-8e39adfd65e8-tls-certificates podName:d3997c3c-be8b-49dc-aeda-8e39adfd65e8 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:47.08689367 +0000 UTC m=+67.946718296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/d3997c3c-be8b-49dc-aeda-8e39adfd65e8-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-59x8f" (UID: "d3997c3c-be8b-49dc-aeda-8e39adfd65e8") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 18:40:46.726785 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:46.726751    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-885b84785-k62rw" event={"ID":"98b10151-b464-420a-95b9-4367a22e5e92","Type":"ContainerStarted","Data":"2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095"}
Sep 17 18:40:46.751068 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:46.751019    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-885b84785-k62rw" podStartSLOduration=1.9425057620000001 podStartE2EDuration="5.751002085s" totalImagesPullingTime="3.808496323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:46.749974402 +0000 UTC m=+67.609799050" watchObservedRunningTime="2026-09-17 18:40:46.751002085 +0000 UTC m=+67.610826732"
Sep 17 18:40:47.090300 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.090266    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d3997c3c-be8b-49dc-aeda-8e39adfd65e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-59x8f\" (UID: \"d3997c3c-be8b-49dc-aeda-8e39adfd65e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"
Sep 17 18:40:47.090444 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.090318    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:47.090499 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.090453    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:47.092915 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.092892    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d3997c3c-be8b-49dc-aeda-8e39adfd65e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-59x8f\" (UID: \"d3997c3c-be8b-49dc-aeda-8e39adfd65e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"
Sep 17 18:40:47.093762 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.093742    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:40:47.093903 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.093884    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:40:47.103253 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.103232    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8969506-84fa-4bcc-a5f6-383cbe2b041f-metrics-certs\") pod \"network-metrics-daemon-vhljw\" (UID: \"e8969506-84fa-4bcc-a5f6-383cbe2b041f\") " pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:47.103253 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.103250    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/670d8bfc-6813-421f-941d-c7e35a57f000-original-pull-secret\") pod \"global-pull-secret-syncer-xnrlh\" (UID: \"670d8bfc-6813-421f-941d-c7e35a57f000\") " pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:47.191091 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.191052    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9vw\" (UniqueName: \"kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw\") pod \"network-check-target-cc2x9\" (UID: \"19bca968-8c00-452e-93fb-8edd90e3f271\") " pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:47.194747 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.194731    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:40:47.209061 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.209040    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:40:47.214518 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.214498    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9vw\" (UniqueName: \"kubernetes.io/projected/19bca968-8c00-452e-93fb-8edd90e3f271-kube-api-access-kd9vw\") pod \"network-check-target-cc2x9\" (UID: \"19bca968-8c00-452e-93fb-8edd90e3f271\") " pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:47.263553 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.263512    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnrlh"
Sep 17 18:40:47.300037 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.299881    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"
Sep 17 18:40:47.366639 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.366614    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7pg8w\""
Sep 17 18:40:47.373148 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.373126    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5jd5p\""
Sep 17 18:40:47.375144 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.375129    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vhljw"
Sep 17 18:40:47.381459 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.381432    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:47.456316 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.456291    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d8df66cc-bt6cg"]
Sep 17 18:40:47.458780 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.458759    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.467767 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.467741    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 18:40:47.476683 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.476659    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d8df66cc-bt6cg"]
Sep 17 18:40:47.493365 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.493333    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-console-config\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.493365 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.493364    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-serving-cert\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.493534 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.493394    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-oauth-config\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.493534 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.493408    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-service-ca\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.493534 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.493423    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-trusted-ca-bundle\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.493534 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.493461    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-oauth-serving-cert\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.493534 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.493521    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzc8r\" (UniqueName: \"kubernetes.io/projected/4e5071d4-786a-4804-b08d-e20372cfd51b-kube-api-access-dzc8r\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.533280 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.533240    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xnrlh"]
Sep 17 18:40:47.535991 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:47.535964    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670d8bfc_6813_421f_941d_c7e35a57f000.slice/crio-55e37d57c8aac0b27f5a43b7024c69fd22176d4dd46338abf23ef9038f8ae4b2 WatchSource:0}: Error finding container 55e37d57c8aac0b27f5a43b7024c69fd22176d4dd46338abf23ef9038f8ae4b2: Status 404 returned error can't find the container with id 55e37d57c8aac0b27f5a43b7024c69fd22176d4dd46338abf23ef9038f8ae4b2
Sep 17 18:40:47.573042 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.572986    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"]
Sep 17 18:40:47.575371 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:47.575345    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3997c3c_be8b_49dc_aeda_8e39adfd65e8.slice/crio-a07cf6be7be5a074a67b94734d64998eac4f30d1ae930b14aea87624b336ec95 WatchSource:0}: Error finding container a07cf6be7be5a074a67b94734d64998eac4f30d1ae930b14aea87624b336ec95: Status 404 returned error can't find the container with id a07cf6be7be5a074a67b94734d64998eac4f30d1ae930b14aea87624b336ec95
Sep 17 18:40:47.594134 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.594108    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-console-config\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.594134 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.594139    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-serving-cert\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.594348 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.594169    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-oauth-config\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.594348 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.594184    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-service-ca\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.594348 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.594200    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-trusted-ca-bundle\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.594348 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.594223    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-oauth-serving-cert\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.594765 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.594444    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzc8r\" (UniqueName: \"kubernetes.io/projected/4e5071d4-786a-4804-b08d-e20372cfd51b-kube-api-access-dzc8r\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.594984 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.594962    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-oauth-serving-cert\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.595142 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.595107    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-console-config\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.595580 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.595555    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-service-ca\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.595667 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.595622    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-trusted-ca-bundle\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.596657 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.596639    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-oauth-config\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.596892 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.596806    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-serving-cert\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.604106 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.604057    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzc8r\" (UniqueName: \"kubernetes.io/projected/4e5071d4-786a-4804-b08d-e20372cfd51b-kube-api-access-dzc8r\") pod \"console-7d8df66cc-bt6cg\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") " pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:47.649105 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.649070    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vhljw"]
Sep 17 18:40:47.651901 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:47.651865    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8969506_84fa_4bcc_a5f6_383cbe2b041f.slice/crio-61801edcbbac92c8b9fd455374eec1252509199c4785064cca16aa27401d432f WatchSource:0}: Error finding container 61801edcbbac92c8b9fd455374eec1252509199c4785064cca16aa27401d432f: Status 404 returned error can't find the container with id 61801edcbbac92c8b9fd455374eec1252509199c4785064cca16aa27401d432f
Sep 17 18:40:47.668379 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.666457    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cc2x9"]
Sep 17 18:40:47.668940 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:47.668908    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bca968_8c00_452e_93fb_8edd90e3f271.slice/crio-4e42e819842d3224d7bf9c4e34e4e68b21f8e9a50292639891caad4adc14be1a WatchSource:0}: Error finding container 4e42e819842d3224d7bf9c4e34e4e68b21f8e9a50292639891caad4adc14be1a: Status 404 returned error can't find the container with id 4e42e819842d3224d7bf9c4e34e4e68b21f8e9a50292639891caad4adc14be1a
Sep 17 18:40:47.731050 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.731014    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cc2x9" event={"ID":"19bca968-8c00-452e-93fb-8edd90e3f271","Type":"ContainerStarted","Data":"4e42e819842d3224d7bf9c4e34e4e68b21f8e9a50292639891caad4adc14be1a"}
Sep 17 18:40:47.732286 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.732256    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vhljw" event={"ID":"e8969506-84fa-4bcc-a5f6-383cbe2b041f","Type":"ContainerStarted","Data":"61801edcbbac92c8b9fd455374eec1252509199c4785064cca16aa27401d432f"}
Sep 17 18:40:47.733377 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.733357    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f" event={"ID":"d3997c3c-be8b-49dc-aeda-8e39adfd65e8","Type":"ContainerStarted","Data":"a07cf6be7be5a074a67b94734d64998eac4f30d1ae930b14aea87624b336ec95"}
Sep 17 18:40:47.734404 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.734384    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xnrlh" event={"ID":"670d8bfc-6813-421f-941d-c7e35a57f000","Type":"ContainerStarted","Data":"55e37d57c8aac0b27f5a43b7024c69fd22176d4dd46338abf23ef9038f8ae4b2"}
Sep 17 18:40:47.769023 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:47.768998    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:48.045164 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:48.045081    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d8df66cc-bt6cg"]
Sep 17 18:40:48.049001 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:48.048969    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5071d4_786a_4804_b08d_e20372cfd51b.slice/crio-7764228add36880462b4320ac6873f4e94a9fa7cef2581371f588bd8ee5f68a1 WatchSource:0}: Error finding container 7764228add36880462b4320ac6873f4e94a9fa7cef2581371f588bd8ee5f68a1: Status 404 returned error can't find the container with id 7764228add36880462b4320ac6873f4e94a9fa7cef2581371f588bd8ee5f68a1
Sep 17 18:40:48.742170 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:48.742127    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8df66cc-bt6cg" event={"ID":"4e5071d4-786a-4804-b08d-e20372cfd51b","Type":"ContainerStarted","Data":"6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2"}
Sep 17 18:40:48.742600 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:48.742194    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8df66cc-bt6cg" event={"ID":"4e5071d4-786a-4804-b08d-e20372cfd51b","Type":"ContainerStarted","Data":"7764228add36880462b4320ac6873f4e94a9fa7cef2581371f588bd8ee5f68a1"}
Sep 17 18:40:48.786863 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:48.786486    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d8df66cc-bt6cg" podStartSLOduration=1.786460881 podStartE2EDuration="1.786460881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:48.783179696 +0000 UTC m=+69.643004370" watchObservedRunningTime="2026-09-17 18:40:48.786460881 +0000 UTC m=+69.646285528"
Sep 17 18:40:49.749270 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:49.749218    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f" event={"ID":"d3997c3c-be8b-49dc-aeda-8e39adfd65e8","Type":"ContainerStarted","Data":"0a58549947d5dec7f8653652a035d2d3b0b9b866be83af99f5b986a65fa203cd"}
Sep 17 18:40:49.750176 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:49.750157    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"
Sep 17 18:40:49.753092 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:49.752979    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vhljw" event={"ID":"e8969506-84fa-4bcc-a5f6-383cbe2b041f","Type":"ContainerStarted","Data":"fabaf103dcf11274520811f8c51463b1859c2b924e3f2997962aa6f110543e94"}
Sep 17 18:40:49.753092 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:49.753035    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vhljw" event={"ID":"e8969506-84fa-4bcc-a5f6-383cbe2b041f","Type":"ContainerStarted","Data":"278fcbb93a975612c8b32980c3efe0ecf17e09cff0d97a8074b133fe6254d4f2"}
Sep 17 18:40:49.755670 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:49.755646    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f"
Sep 17 18:40:49.776545 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:49.776488    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-59x8f" podStartSLOduration=2.176522484 podStartE2EDuration="3.776472387s" totalImagesPullingTime="1.599949903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:49.776160074 +0000 UTC m=+70.635984742" watchObservedRunningTime="2026-09-17 18:40:49.776472387 +0000 UTC m=+70.636297034"
Sep 17 18:40:49.808796 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:49.808746    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vhljw" podStartSLOduration=67.2838063 podStartE2EDuration="1m8.808732595s" totalImagesPullingTime="1.524926295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:49.807204434 +0000 UTC m=+70.667029101" watchObservedRunningTime="2026-09-17 18:40:49.808732595 +0000 UTC m=+70.668557287"
Sep 17 18:40:50.710501 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:50.710367    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tx5nx"
Sep 17 18:40:52.190865 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:52.190439    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:52.190865 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:52.190510    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:52.196430 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:52.196402    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:52.766338 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:52.766245    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cc2x9" event={"ID":"19bca968-8c00-452e-93fb-8edd90e3f271","Type":"ContainerStarted","Data":"fbaf3798aef8fcd812df823bfaeee1f8cca5598b24227c0c316aa11fda851742"}
Sep 17 18:40:52.766338 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:52.766297    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:40:52.767772 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:52.767744    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xnrlh" event={"ID":"670d8bfc-6813-421f-941d-c7e35a57f000","Type":"ContainerStarted","Data":"7110de2257978216cd3e50d678edf886787ec4de621d22d6e91c0e6cedbca858"}
Sep 17 18:40:52.771515 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:52.771495    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:40:52.819785 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:52.819730    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cc2x9" podStartSLOduration=67.086177841 podStartE2EDuration="1m11.819686815s" totalImagesPullingTime="4.733508974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:52.788381238 +0000 UTC m=+73.648205886" watchObservedRunningTime="2026-09-17 18:40:52.819686815 +0000 UTC m=+73.679511471"
Sep 17 18:40:52.841802 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:52.841753    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xnrlh" podStartSLOduration=66.971998077 podStartE2EDuration="1m11.841737548s" totalImagesPullingTime="4.869739471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:52.841660359 +0000 UTC m=+73.701485007" watchObservedRunningTime="2026-09-17 18:40:52.841737548 +0000 UTC m=+73.701562198"
Sep 17 18:40:55.062857 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.062809    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"]
Sep 17 18:40:55.065002 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.064986    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.068253 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.068228    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:40:55.068253 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.068240    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 18:40:55.068430 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.068240    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:40:55.069088 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.069070    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 18:40:55.069185 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.069138    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8grp9\""
Sep 17 18:40:55.071796 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.071779    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:40:55.086337 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.086313    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"]
Sep 17 18:40:55.088609 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.088589    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"]
Sep 17 18:40:55.088703 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.088691    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.095276 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.095253    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 18:40:55.095469 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.095452    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7x9c5\""
Sep 17 18:40:55.096035 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.096019    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 18:40:55.096094 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.096074    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 18:40:55.103591 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.103572    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"]
Sep 17 18:40:55.144307 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.144273    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nzlvr"]
Sep 17 18:40:55.146591 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.146575    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.150362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.150342    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:40:55.150465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.150397    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bbjrk\""
Sep 17 18:40:55.151630 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151594    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7a9e79-81ac-4cfd-a067-33df41345c65-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.151750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151637    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.151750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151678    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.151750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151704    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twl2j\" (UniqueName: \"kubernetes.io/projected/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-api-access-twl2j\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.151750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151737    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f7a9e79-81ac-4cfd-a067-33df41345c65-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.151995 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151762    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.151995 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151794    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f7a9e79-81ac-4cfd-a067-33df41345c65-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.151995 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151819    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.151995 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151860    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:40:55.151995 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151897    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.151995 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151911    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:40:55.151995 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.151924    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zsb7\" (UniqueName: \"kubernetes.io/projected/6f7a9e79-81ac-4cfd-a067-33df41345c65-kube-api-access-5zsb7\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.252547 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252519    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df5b11cf-b032-4878-b86d-af0b9731236a-root\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.252547 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252549    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-tls\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.252767 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252575    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.252767 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252631    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zsb7\" (UniqueName: \"kubernetes.io/projected/6f7a9e79-81ac-4cfd-a067-33df41345c65-kube-api-access-5zsb7\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.252767 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252659    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df5b11cf-b032-4878-b86d-af0b9731236a-metrics-client-ca\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.252767 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252678    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgm8l\" (UniqueName: \"kubernetes.io/projected/df5b11cf-b032-4878-b86d-af0b9731236a-kube-api-access-lgm8l\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.252767 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252706    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.252767 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252733    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7a9e79-81ac-4cfd-a067-33df41345c65-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.252767 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252751    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.253122 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252777    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df5b11cf-b032-4878-b86d-af0b9731236a-sys\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.253122 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252808    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.253122 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252860    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twl2j\" (UniqueName: \"kubernetes.io/projected/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-api-access-twl2j\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.253122 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252891    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-wtmp\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.253122 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252925    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f7a9e79-81ac-4cfd-a067-33df41345c65-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.253122 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.252950    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.253384 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.253190    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.253384 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.253213    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-textfile\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.253384 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:55.253284    2586 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 18:40:55.253384 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.253309    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.253552 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:55.253397    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-tls podName:0c0ac57d-1215-47ce-88ab-f82dd383dc4f nodeName:}" failed. No retries permitted until 2026-09-17 18:40:55.753323661 +0000 UTC m=+76.613148287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-b5vzc" (UID: "0c0ac57d-1215-47ce-88ab-f82dd383dc4f") : secret "kube-state-metrics-tls" not found
Sep 17 18:40:55.253552 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.253496    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7a9e79-81ac-4cfd-a067-33df41345c65-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.253662 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.253554    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f7a9e79-81ac-4cfd-a067-33df41345c65-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.253662 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.253581    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.253759 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.253699    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.254089 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.254067    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.255428 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.255399    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.255566 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.255551    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f7a9e79-81ac-4cfd-a067-33df41345c65-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.256011 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.255993    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f7a9e79-81ac-4cfd-a067-33df41345c65-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.263790 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.263764    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zsb7\" (UniqueName: \"kubernetes.io/projected/6f7a9e79-81ac-4cfd-a067-33df41345c65-kube-api-access-5zsb7\") pod \"openshift-state-metrics-86c5bc79d6-6twll\" (UID: \"6f7a9e79-81ac-4cfd-a067-33df41345c65\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.268513 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.268490    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twl2j\" (UniqueName: \"kubernetes.io/projected/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-api-access-twl2j\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.354405 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354376    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df5b11cf-b032-4878-b86d-af0b9731236a-root\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.354405 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354409    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-tls\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.354647 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354439    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df5b11cf-b032-4878-b86d-af0b9731236a-metrics-client-ca\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.354647 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354484    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgm8l\" (UniqueName: \"kubernetes.io/projected/df5b11cf-b032-4878-b86d-af0b9731236a-kube-api-access-lgm8l\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.354647 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354488    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df5b11cf-b032-4878-b86d-af0b9731236a-root\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.354647 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354531    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.354647 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:55.354557    2586 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 18:40:55.354647 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354603    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df5b11cf-b032-4878-b86d-af0b9731236a-sys\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.354647 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354568    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df5b11cf-b032-4878-b86d-af0b9731236a-sys\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.354647 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:40:55.354629    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-tls podName:df5b11cf-b032-4878-b86d-af0b9731236a nodeName:}" failed. No retries permitted until 2026-09-17 18:40:55.854606896 +0000 UTC m=+76.714431534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-tls") pod "node-exporter-nzlvr" (UID: "df5b11cf-b032-4878-b86d-af0b9731236a") : secret "node-exporter-tls" not found
Sep 17 18:40:55.355029 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354687    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-wtmp\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.355029 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354736    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-textfile\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.355029 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354766    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.355029 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.354927    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-wtmp\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.355160 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.355089    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df5b11cf-b032-4878-b86d-af0b9731236a-metrics-client-ca\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.355160 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.355111    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-textfile\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.355226 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.355167    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.357211 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.357190    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.368616 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.368597    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgm8l\" (UniqueName: \"kubernetes.io/projected/df5b11cf-b032-4878-b86d-af0b9731236a-kube-api-access-lgm8l\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.373510 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.373495    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"
Sep 17 18:40:55.643743 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.643718    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll"]
Sep 17 18:40:55.646121 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:55.646091    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7a9e79_81ac_4cfd_a067_33df41345c65.slice/crio-fbbb7124509287f60d79594e28e65bb5ffa1ed74ac2113d0b8d763a464c64377 WatchSource:0}: Error finding container fbbb7124509287f60d79594e28e65bb5ffa1ed74ac2113d0b8d763a464c64377: Status 404 returned error can't find the container with id fbbb7124509287f60d79594e28e65bb5ffa1ed74ac2113d0b8d763a464c64377
Sep 17 18:40:55.757261 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.757233    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.759608 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.759586    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0ac57d-1215-47ce-88ab-f82dd383dc4f-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-b5vzc\" (UID: \"0c0ac57d-1215-47ce-88ab-f82dd383dc4f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:55.780422 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.780381    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll" event={"ID":"6f7a9e79-81ac-4cfd-a067-33df41345c65","Type":"ContainerStarted","Data":"654e01e50f12bc38b831bd950d4ceb9fa09d951c95f97a326756ea0fd94a4219"}
Sep 17 18:40:55.780592 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.780462    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll" event={"ID":"6f7a9e79-81ac-4cfd-a067-33df41345c65","Type":"ContainerStarted","Data":"763c03bada36867fa490f56929cb6543a899c58b0f4e8a2a2c5c8464f92e87e2"}
Sep 17 18:40:55.780592 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.780496    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll" event={"ID":"6f7a9e79-81ac-4cfd-a067-33df41345c65","Type":"ContainerStarted","Data":"fbbb7124509287f60d79594e28e65bb5ffa1ed74ac2113d0b8d763a464c64377"}
Sep 17 18:40:55.858606 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.858562    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-tls\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.861033 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.861014    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df5b11cf-b032-4878-b86d-af0b9731236a-node-exporter-tls\") pod \"node-exporter-nzlvr\" (UID: \"df5b11cf-b032-4878-b86d-af0b9731236a\") " pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:55.997243 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:55.997153    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"
Sep 17 18:40:56.055506 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:56.055475    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nzlvr"
Sep 17 18:40:56.065865 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:56.065817    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5b11cf_b032_4878_b86d_af0b9731236a.slice/crio-89f01cfa1776ca65051de95bca3d9da6018d70ee6e8ce419b9b77eca38b4e756 WatchSource:0}: Error finding container 89f01cfa1776ca65051de95bca3d9da6018d70ee6e8ce419b9b77eca38b4e756: Status 404 returned error can't find the container with id 89f01cfa1776ca65051de95bca3d9da6018d70ee6e8ce419b9b77eca38b4e756
Sep 17 18:40:56.273392 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:56.273322    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-b5vzc"]
Sep 17 18:40:56.278222 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:56.278185    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0ac57d_1215_47ce_88ab_f82dd383dc4f.slice/crio-5207123a482f2969b88d5549fed0554c8f210f586596433b43b0abebbd560075 WatchSource:0}: Error finding container 5207123a482f2969b88d5549fed0554c8f210f586596433b43b0abebbd560075: Status 404 returned error can't find the container with id 5207123a482f2969b88d5549fed0554c8f210f586596433b43b0abebbd560075
Sep 17 18:40:56.786748 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:56.786703    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzlvr" event={"ID":"df5b11cf-b032-4878-b86d-af0b9731236a","Type":"ContainerStarted","Data":"89f01cfa1776ca65051de95bca3d9da6018d70ee6e8ce419b9b77eca38b4e756"}
Sep 17 18:40:56.788909 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:56.788873    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc" event={"ID":"0c0ac57d-1215-47ce-88ab-f82dd383dc4f","Type":"ContainerStarted","Data":"5207123a482f2969b88d5549fed0554c8f210f586596433b43b0abebbd560075"}
Sep 17 18:40:57.770197 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.770091    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:57.770197 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.770171    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:57.776267 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.776238    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:57.794967 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.794934    2586 generic.go:356] "Generic (PLEG): container finished" podID="df5b11cf-b032-4878-b86d-af0b9731236a" containerID="84d86a5714dee8c645aa4a7878ad83862a03b3d4056122109a5b1efea10626a7" exitCode=0
Sep 17 18:40:57.795128 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.795014    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzlvr" event={"ID":"df5b11cf-b032-4878-b86d-af0b9731236a","Type":"ContainerDied","Data":"84d86a5714dee8c645aa4a7878ad83862a03b3d4056122109a5b1efea10626a7"}
Sep 17 18:40:57.797609 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.797551    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll" event={"ID":"6f7a9e79-81ac-4cfd-a067-33df41345c65","Type":"ContainerStarted","Data":"44e6d9741267ac337bc2fc930ce919c359a2a9ddcdf427bbea0d40d0ecee84f8"}
Sep 17 18:40:57.801508 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.801471    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc" event={"ID":"0c0ac57d-1215-47ce-88ab-f82dd383dc4f","Type":"ContainerStarted","Data":"3408ede58497259208953f5dcdfac4ff1c1b356a9df5a0623b42b60bf8e7e92f"}
Sep 17 18:40:57.801608 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.801527    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc" event={"ID":"0c0ac57d-1215-47ce-88ab-f82dd383dc4f","Type":"ContainerStarted","Data":"bb9e21acb492c1858b95b0fdb4254c4076cb87299b2363bd2d3dcb5e08902af3"}
Sep 17 18:40:57.801608 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.801556    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc" event={"ID":"0c0ac57d-1215-47ce-88ab-f82dd383dc4f","Type":"ContainerStarted","Data":"89c752a78f432dfec765c9bfe79f667455a119796b953da2d5c4297e509dc3d7"}
Sep 17 18:40:57.806138 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.806119    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:40:57.849432 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.849372    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6twll" podStartSLOduration=1.9123271910000001 podStartE2EDuration="2.849350361s" totalImagesPullingTime="937.02317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:57.847394147 +0000 UTC m=+78.707218828" watchObservedRunningTime="2026-09-17 18:40:57.849350361 +0000 UTC m=+78.709175011"
Sep 17 18:40:57.892880 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.892807    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-b5vzc" podStartSLOduration=1.7247462 podStartE2EDuration="2.892787176s" totalImagesPullingTime="1.168040976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:57.891756793 +0000 UTC m=+78.751581466" watchObservedRunningTime="2026-09-17 18:40:57.892787176 +0000 UTC m=+78.752611824"
Sep 17 18:40:57.919081 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:57.919041    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-885b84785-k62rw"]
Sep 17 18:40:58.229953 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.229921    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"]
Sep 17 18:40:58.234129 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.234096    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.237284 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.237258    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 18:40:58.237788 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.237767    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dcabqlcatk54e\""
Sep 17 18:40:58.238067 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.238051    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 18:40:58.238133 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.238102    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 18:40:58.239192 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.239178    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mvj8j\""
Sep 17 18:40:58.239192 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.239188    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 18:40:58.240303 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.240286    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 18:40:58.251224 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.251201    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"]
Sep 17 18:40:58.278658 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.278631    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3855f0a-c397-410b-a52c-69cef688a2b1-metrics-client-ca\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.278789 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.278671    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-tls\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.278789 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.278693    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.278789 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.278709    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.278789 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.278731    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.278974 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.278792    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbrs\" (UniqueName: \"kubernetes.io/projected/e3855f0a-c397-410b-a52c-69cef688a2b1-kube-api-access-qlbrs\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.278974 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.278809    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.278974 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.278916    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-grpc-tls\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.379991 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.379952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-grpc-tls\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.379991 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.379995    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3855f0a-c397-410b-a52c-69cef688a2b1-metrics-client-ca\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.380194 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.380015    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-tls\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.380194 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.380036    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.380194 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.380179    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.380344 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.380239    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.380344 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.380287    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbrs\" (UniqueName: \"kubernetes.io/projected/e3855f0a-c397-410b-a52c-69cef688a2b1-kube-api-access-qlbrs\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.380344 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.380319    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.380905 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.380878    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3855f0a-c397-410b-a52c-69cef688a2b1-metrics-client-ca\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.382995 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.382969    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.383088 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.382970    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.383194 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.383173    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.383372 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.383358    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.383404 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.383364    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-thanos-querier-tls\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.383434 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.383407    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3855f0a-c397-410b-a52c-69cef688a2b1-secret-grpc-tls\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.388524 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.388508    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbrs\" (UniqueName: \"kubernetes.io/projected/e3855f0a-c397-410b-a52c-69cef688a2b1-kube-api-access-qlbrs\") pod \"thanos-querier-54d8d5b55c-pddhs\" (UID: \"e3855f0a-c397-410b-a52c-69cef688a2b1\") " pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.543614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.543522    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:40:58.807246 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.807153    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzlvr" event={"ID":"df5b11cf-b032-4878-b86d-af0b9731236a","Type":"ContainerStarted","Data":"130c28b08c46205917645c061d3b833188492fb0bb08cccb5ff40ee806404772"}
Sep 17 18:40:58.807246 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.807214    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzlvr" event={"ID":"df5b11cf-b032-4878-b86d-af0b9731236a","Type":"ContainerStarted","Data":"6cf08e4c9e8dce3ecbcc51409c44dc756bd16392a9537a81f82a9fabdd2fa242"}
Sep 17 18:40:58.830404 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.830382    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"]
Sep 17 18:40:58.832395 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:40:58.832371    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3855f0a_c397_410b_a52c_69cef688a2b1.slice/crio-5ac31768c2a32273e269cb3a3b74c8f6fe9eb5ae189df8e9a6c79a931bcbb50e WatchSource:0}: Error finding container 5ac31768c2a32273e269cb3a3b74c8f6fe9eb5ae189df8e9a6c79a931bcbb50e: Status 404 returned error can't find the container with id 5ac31768c2a32273e269cb3a3b74c8f6fe9eb5ae189df8e9a6c79a931bcbb50e
Sep 17 18:40:58.835303 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:58.835265    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nzlvr" podStartSLOduration=2.879523796 podStartE2EDuration="3.835250022s" totalImagesPullingTime="878.784276ms" totalInitContainerRuntime="76.94195ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:58.833811821 +0000 UTC m=+79.693636492" watchObservedRunningTime="2026-09-17 18:40:58.835250022 +0000 UTC m=+79.695074666"
Sep 17 18:40:59.811433 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:40:59.811398    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs" event={"ID":"e3855f0a-c397-410b-a52c-69cef688a2b1","Type":"ContainerStarted","Data":"5ac31768c2a32273e269cb3a3b74c8f6fe9eb5ae189df8e9a6c79a931bcbb50e"}
Sep 17 18:41:00.229871 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.229812    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fd7cf5f7b-55wgr"]
Sep 17 18:41:00.232351 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.232329    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.298155 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.298116    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-serving-cert\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.298155 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.298168    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-oauth-serving-cert\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.298375 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.298199    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-trusted-ca-bundle\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.298375 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.298240    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-service-ca\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.298375 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.298275    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwhjn\" (UniqueName: \"kubernetes.io/projected/e34a19e1-5079-4b21-90d4-61977a8597c4-kube-api-access-hwhjn\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.298375 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.298313    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-oauth-config\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.298375 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.298346    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-console-config\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.307618 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.307581    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fd7cf5f7b-55wgr"]
Sep 17 18:41:00.399252 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.399215    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-console-config\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.399414 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.399277    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-serving-cert\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.399414 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.399312    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-oauth-serving-cert\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.399414 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.399341    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-trusted-ca-bundle\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.399414 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.399373    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-service-ca\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.399633 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.399440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwhjn\" (UniqueName: \"kubernetes.io/projected/e34a19e1-5079-4b21-90d4-61977a8597c4-kube-api-access-hwhjn\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.399633 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.399510    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-oauth-config\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.400132 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.400099    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-oauth-serving-cert\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.400132 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.400123    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-console-config\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.400314 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.400186    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-service-ca\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.400605 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.400569    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-trusted-ca-bundle\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.402572 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.402547    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-oauth-config\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.402964 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.402944    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-serving-cert\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.409608 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.409587    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwhjn\" (UniqueName: \"kubernetes.io/projected/e34a19e1-5079-4b21-90d4-61977a8597c4-kube-api-access-hwhjn\") pod \"console-7fd7cf5f7b-55wgr\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") " pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.543167 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.543064    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:00.816529 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.816487    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs" event={"ID":"e3855f0a-c397-410b-a52c-69cef688a2b1","Type":"ContainerStarted","Data":"823c5cd507beb174f8b71ce8b07e34217a1fe54423df00d95bbdec525f7480a0"}
Sep 17 18:41:00.959481 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:00.959346    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fd7cf5f7b-55wgr"]
Sep 17 18:41:00.961877 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:41:00.961852    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34a19e1_5079_4b21_90d4_61977a8597c4.slice/crio-cb3966aa113d6ce534ef404e51ff845b0283f0805d90ae6f003485c3caeb256e WatchSource:0}: Error finding container cb3966aa113d6ce534ef404e51ff845b0283f0805d90ae6f003485c3caeb256e: Status 404 returned error can't find the container with id cb3966aa113d6ce534ef404e51ff845b0283f0805d90ae6f003485c3caeb256e
Sep 17 18:41:01.824076 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:01.824039    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs" event={"ID":"e3855f0a-c397-410b-a52c-69cef688a2b1","Type":"ContainerStarted","Data":"5cca61f144d69538c28adea5a7cbc94ccafd3ea96027cff70f4526c4eaad5d7a"}
Sep 17 18:41:01.824529 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:01.824115    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs" event={"ID":"e3855f0a-c397-410b-a52c-69cef688a2b1","Type":"ContainerStarted","Data":"e94e57c5b850e6d4690843ca2029d9884242abae6aef3fc6c932b6984e1bb176"}
Sep 17 18:41:01.824529 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:01.824160    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs" event={"ID":"e3855f0a-c397-410b-a52c-69cef688a2b1","Type":"ContainerStarted","Data":"56e0320ed0c4abdb2e40fef2c981a380c8644a6bd3077c03a065b10b00a4b9dd"}
Sep 17 18:41:01.824529 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:01.824201    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:41:01.824529 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:01.824227    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs" event={"ID":"e3855f0a-c397-410b-a52c-69cef688a2b1","Type":"ContainerStarted","Data":"93c7f1e07f3be938da77bb51735aa9bf8fcce3da322301f7ac3a6f3995c31c51"}
Sep 17 18:41:01.824529 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:01.824265    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs" event={"ID":"e3855f0a-c397-410b-a52c-69cef688a2b1","Type":"ContainerStarted","Data":"04570d4429e9a9eef56f91d5761fe55aa2e3f78ff8b490b0a20f03b67c9cbbd4"}
Sep 17 18:41:01.825691 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:01.825668    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd7cf5f7b-55wgr" event={"ID":"e34a19e1-5079-4b21-90d4-61977a8597c4","Type":"ContainerStarted","Data":"32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d"}
Sep 17 18:41:01.825807 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:01.825715    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd7cf5f7b-55wgr" event={"ID":"e34a19e1-5079-4b21-90d4-61977a8597c4","Type":"ContainerStarted","Data":"cb3966aa113d6ce534ef404e51ff845b0283f0805d90ae6f003485c3caeb256e"}
Sep 17 18:41:01.855388 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:01.855337    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs" podStartSLOduration=1.335692656 podStartE2EDuration="3.855322325s" totalImagesPullingTime="2.519629669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:01.853252485 +0000 UTC m=+82.713077151" watchObservedRunningTime="2026-09-17 18:41:01.855322325 +0000 UTC m=+82.715146973"
Sep 17 18:41:01.879685 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:01.879634    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fd7cf5f7b-55wgr" podStartSLOduration=1.8796103880000001 podStartE2EDuration="1.879610388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:01.878673979 +0000 UTC m=+82.738498626" watchObservedRunningTime="2026-09-17 18:41:01.879610388 +0000 UTC m=+82.739435036"
Sep 17 18:41:07.835334 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:07.835305    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-54d8d5b55c-pddhs"
Sep 17 18:41:10.544122 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:10.544090    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:10.544544 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:10.544364    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:10.549056 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:10.549037    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:10.861327 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:10.861303    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:41:10.949942 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:10.949915    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d8df66cc-bt6cg"]
Sep 17 18:41:22.939983 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:22.939931    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-885b84785-k62rw" podUID="98b10151-b464-420a-95b9-4367a22e5e92" containerName="console" containerID="cri-o://2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095" gracePeriod=15
Sep 17 18:41:23.708875 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.708809    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:41:23.777247 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.777220    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cc2x9"
Sep 17 18:41:23.802180 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.802147    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-serving-cert\") pod \"98b10151-b464-420a-95b9-4367a22e5e92\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") "
Sep 17 18:41:23.802180 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.802184    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-oauth-serving-cert\") pod \"98b10151-b464-420a-95b9-4367a22e5e92\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") "
Sep 17 18:41:23.802409 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.802232    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpwh4\" (UniqueName: \"kubernetes.io/projected/98b10151-b464-420a-95b9-4367a22e5e92-kube-api-access-kpwh4\") pod \"98b10151-b464-420a-95b9-4367a22e5e92\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") "
Sep 17 18:41:23.802409 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.802254    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-oauth-config\") pod \"98b10151-b464-420a-95b9-4367a22e5e92\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") "
Sep 17 18:41:23.802409 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.802279    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-service-ca\") pod \"98b10151-b464-420a-95b9-4367a22e5e92\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") "
Sep 17 18:41:23.802409 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.802314    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-console-config\") pod \"98b10151-b464-420a-95b9-4367a22e5e92\" (UID: \"98b10151-b464-420a-95b9-4367a22e5e92\") "
Sep 17 18:41:23.802740 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.802714    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "98b10151-b464-420a-95b9-4367a22e5e92" (UID: "98b10151-b464-420a-95b9-4367a22e5e92"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:23.802877 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.802728    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-service-ca" (OuterVolumeSpecName: "service-ca") pod "98b10151-b464-420a-95b9-4367a22e5e92" (UID: "98b10151-b464-420a-95b9-4367a22e5e92"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:23.802877 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.802756    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-console-config" (OuterVolumeSpecName: "console-config") pod "98b10151-b464-420a-95b9-4367a22e5e92" (UID: "98b10151-b464-420a-95b9-4367a22e5e92"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:23.804776 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.804748    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "98b10151-b464-420a-95b9-4367a22e5e92" (UID: "98b10151-b464-420a-95b9-4367a22e5e92"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:23.805034 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.805007    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "98b10151-b464-420a-95b9-4367a22e5e92" (UID: "98b10151-b464-420a-95b9-4367a22e5e92"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:23.805085 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.805039    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b10151-b464-420a-95b9-4367a22e5e92-kube-api-access-kpwh4" (OuterVolumeSpecName: "kube-api-access-kpwh4") pod "98b10151-b464-420a-95b9-4367a22e5e92" (UID: "98b10151-b464-420a-95b9-4367a22e5e92"). InnerVolumeSpecName "kube-api-access-kpwh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:23.903115 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.903030    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpwh4\" (UniqueName: \"kubernetes.io/projected/98b10151-b464-420a-95b9-4367a22e5e92-kube-api-access-kpwh4\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:23.903115 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.903061    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-oauth-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:23.903115 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.903075    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:23.903115 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.903088    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-console-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:23.903115 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.903100    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98b10151-b464-420a-95b9-4367a22e5e92-console-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:23.903115 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.903110    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98b10151-b464-420a-95b9-4367a22e5e92-oauth-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:23.906214 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.906190    2586 generic.go:356] "Generic (PLEG): container finished" podID="98b10151-b464-420a-95b9-4367a22e5e92" containerID="2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095" exitCode=0
Sep 17 18:41:23.906323 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.906235    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-885b84785-k62rw"
Sep 17 18:41:23.906323 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.906262    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-885b84785-k62rw" event={"ID":"98b10151-b464-420a-95b9-4367a22e5e92","Type":"ContainerDied","Data":"2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095"}
Sep 17 18:41:23.906323 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.906299    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-885b84785-k62rw" event={"ID":"98b10151-b464-420a-95b9-4367a22e5e92","Type":"ContainerDied","Data":"c96a7b4fd19e252664a217024befb2d625c016bc326187a080bffedffda3f757"}
Sep 17 18:41:23.906323 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.906318    2586 scope.go:117] "RemoveContainer" containerID="2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095"
Sep 17 18:41:23.922720 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.922700    2586 scope.go:117] "RemoveContainer" containerID="2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095"
Sep 17 18:41:23.923027 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:41:23.922996    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095\": container with ID starting with 2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095 not found: ID does not exist" containerID="2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095"
Sep 17 18:41:23.923087 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.923036    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095"} err="failed to get container status \"2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095\": rpc error: code = NotFound desc = could not find container \"2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095\": container with ID starting with 2f3317e5abc707376744e6b96a1cf0d6115f70b13d6aa81a98c00d58ab57b095 not found: ID does not exist"
Sep 17 18:41:23.934604 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.934579    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-885b84785-k62rw"]
Sep 17 18:41:23.940712 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:23.940688    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-885b84785-k62rw"]
Sep 17 18:41:25.457428 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:25.457393    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98b10151-b464-420a-95b9-4367a22e5e92" path="/var/lib/kubelet/pods/98b10151-b464-420a-95b9-4367a22e5e92/volumes"
Sep 17 18:41:35.969032 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:35.968989    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d8df66cc-bt6cg" podUID="4e5071d4-786a-4804-b08d-e20372cfd51b" containerName="console" containerID="cri-o://6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2" gracePeriod=15
Sep 17 18:41:36.223035 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.222976    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:41:36.299826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.299791    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-oauth-config\") pod \"4e5071d4-786a-4804-b08d-e20372cfd51b\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") "
Sep 17 18:41:36.299826 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.299824    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-oauth-serving-cert\") pod \"4e5071d4-786a-4804-b08d-e20372cfd51b\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") "
Sep 17 18:41:36.300063 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.299883    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-console-config\") pod \"4e5071d4-786a-4804-b08d-e20372cfd51b\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") "
Sep 17 18:41:36.300063 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.299900    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-serving-cert\") pod \"4e5071d4-786a-4804-b08d-e20372cfd51b\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") "
Sep 17 18:41:36.300063 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.299922    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-service-ca\") pod \"4e5071d4-786a-4804-b08d-e20372cfd51b\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") "
Sep 17 18:41:36.300301 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.300272    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-console-config" (OuterVolumeSpecName: "console-config") pod "4e5071d4-786a-4804-b08d-e20372cfd51b" (UID: "4e5071d4-786a-4804-b08d-e20372cfd51b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:36.300357 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.300280    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4e5071d4-786a-4804-b08d-e20372cfd51b" (UID: "4e5071d4-786a-4804-b08d-e20372cfd51b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:36.300401 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.300354    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-service-ca" (OuterVolumeSpecName: "service-ca") pod "4e5071d4-786a-4804-b08d-e20372cfd51b" (UID: "4e5071d4-786a-4804-b08d-e20372cfd51b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:36.302252 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.302217    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4e5071d4-786a-4804-b08d-e20372cfd51b" (UID: "4e5071d4-786a-4804-b08d-e20372cfd51b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:36.302345 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.302278    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4e5071d4-786a-4804-b08d-e20372cfd51b" (UID: "4e5071d4-786a-4804-b08d-e20372cfd51b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:36.400649 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.400614    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzc8r\" (UniqueName: \"kubernetes.io/projected/4e5071d4-786a-4804-b08d-e20372cfd51b-kube-api-access-dzc8r\") pod \"4e5071d4-786a-4804-b08d-e20372cfd51b\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") "
Sep 17 18:41:36.400811 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.400663    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-trusted-ca-bundle\") pod \"4e5071d4-786a-4804-b08d-e20372cfd51b\" (UID: \"4e5071d4-786a-4804-b08d-e20372cfd51b\") "
Sep 17 18:41:36.401051 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.400945    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-oauth-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:36.401051 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.400972    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-console-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:36.401051 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.400986    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:36.401051 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.400998    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:36.401051 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.401012    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e5071d4-786a-4804-b08d-e20372cfd51b-console-oauth-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:36.401051 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.401019    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4e5071d4-786a-4804-b08d-e20372cfd51b" (UID: "4e5071d4-786a-4804-b08d-e20372cfd51b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:36.402905 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.402885    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5071d4-786a-4804-b08d-e20372cfd51b-kube-api-access-dzc8r" (OuterVolumeSpecName: "kube-api-access-dzc8r") pod "4e5071d4-786a-4804-b08d-e20372cfd51b" (UID: "4e5071d4-786a-4804-b08d-e20372cfd51b"). InnerVolumeSpecName "kube-api-access-dzc8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:36.502273 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.502189    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzc8r\" (UniqueName: \"kubernetes.io/projected/4e5071d4-786a-4804-b08d-e20372cfd51b-kube-api-access-dzc8r\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:36.502273 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.502228    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e5071d4-786a-4804-b08d-e20372cfd51b-trusted-ca-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:41:36.958788 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.958753    2586 generic.go:356] "Generic (PLEG): container finished" podID="4e5071d4-786a-4804-b08d-e20372cfd51b" containerID="6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2" exitCode=0
Sep 17 18:41:36.958989 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.958798    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8df66cc-bt6cg"
Sep 17 18:41:36.958989 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.958860    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8df66cc-bt6cg" event={"ID":"4e5071d4-786a-4804-b08d-e20372cfd51b","Type":"ContainerDied","Data":"6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2"}
Sep 17 18:41:36.958989 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.958918    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8df66cc-bt6cg" event={"ID":"4e5071d4-786a-4804-b08d-e20372cfd51b","Type":"ContainerDied","Data":"7764228add36880462b4320ac6873f4e94a9fa7cef2581371f588bd8ee5f68a1"}
Sep 17 18:41:36.958989 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.958940    2586 scope.go:117] "RemoveContainer" containerID="6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2"
Sep 17 18:41:36.968669 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.968643    2586 scope.go:117] "RemoveContainer" containerID="6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2"
Sep 17 18:41:36.969075 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:41:36.969048    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2\": container with ID starting with 6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2 not found: ID does not exist" containerID="6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2"
Sep 17 18:41:36.969465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.969090    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2"} err="failed to get container status \"6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2\": rpc error: code = NotFound desc = could not find container \"6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2\": container with ID starting with 6725f19993cc5e79cc44623364eb8addba231cd5491728f2b2a70569c626e7d2 not found: ID does not exist"
Sep 17 18:41:36.983742 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.983719    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d8df66cc-bt6cg"]
Sep 17 18:41:36.986372 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:36.986349    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d8df66cc-bt6cg"]
Sep 17 18:41:37.456583 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:41:37.456539    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e5071d4-786a-4804-b08d-e20372cfd51b" path="/var/lib/kubelet/pods/4e5071d4-786a-4804-b08d-e20372cfd51b/volumes"
Sep 17 18:42:19.690620 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.690581    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"]
Sep 17 18:42:19.691060 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.690864    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5071d4-786a-4804-b08d-e20372cfd51b" containerName="console"
Sep 17 18:42:19.691060 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.690877    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5071d4-786a-4804-b08d-e20372cfd51b" containerName="console"
Sep 17 18:42:19.691060 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.690885    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b10151-b464-420a-95b9-4367a22e5e92" containerName="console"
Sep 17 18:42:19.691060 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.690891    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b10151-b464-420a-95b9-4367a22e5e92" containerName="console"
Sep 17 18:42:19.691060 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.690934    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b10151-b464-420a-95b9-4367a22e5e92" containerName="console"
Sep 17 18:42:19.691060 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.690941    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5071d4-786a-4804-b08d-e20372cfd51b" containerName="console"
Sep 17 18:42:19.692861 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.692825    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.702549 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.702529    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 18:42:19.702654 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.702586    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wm9xh\""
Sep 17 18:42:19.702827 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.702793    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 18:42:19.703274 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.703260    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 18:42:19.703898 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.703885    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 18:42:19.709105 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.709089    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 18:42:19.716180 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.716159    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 18:42:19.730025 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.730003    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"]
Sep 17 18:42:19.815137 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.815095    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.815137 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.815138    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9030b38c-0472-4cd4-bc5d-65371412f054-serving-certs-ca-bundle\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.815356 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.815161    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-secret-telemeter-client\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.815356 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.815182    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9030b38c-0472-4cd4-bc5d-65371412f054-metrics-client-ca\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.815356 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.815208    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9030b38c-0472-4cd4-bc5d-65371412f054-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.815356 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.815228    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvnb2\" (UniqueName: \"kubernetes.io/projected/9030b38c-0472-4cd4-bc5d-65371412f054-kube-api-access-rvnb2\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.815356 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.815287    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-telemeter-client-tls\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.815356 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.815305    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-federate-client-tls\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.915628 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.915579    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9030b38c-0472-4cd4-bc5d-65371412f054-serving-certs-ca-bundle\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.915628 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.915632    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-secret-telemeter-client\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.915874 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.915751    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9030b38c-0472-4cd4-bc5d-65371412f054-metrics-client-ca\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.915874 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.915791    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9030b38c-0472-4cd4-bc5d-65371412f054-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.915874 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.915818    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvnb2\" (UniqueName: \"kubernetes.io/projected/9030b38c-0472-4cd4-bc5d-65371412f054-kube-api-access-rvnb2\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.916041 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.915889    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-telemeter-client-tls\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.916041 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.915910    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-federate-client-tls\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.916041 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.915950    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.916376 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.916351    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9030b38c-0472-4cd4-bc5d-65371412f054-serving-certs-ca-bundle\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.916964 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.916938    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9030b38c-0472-4cd4-bc5d-65371412f054-metrics-client-ca\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.917086 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.917063    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9030b38c-0472-4cd4-bc5d-65371412f054-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.918462 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.918434    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.918562 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.918502    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-federate-client-tls\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.918562 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.918524    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-telemeter-client-tls\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.918562 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.918544    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9030b38c-0472-4cd4-bc5d-65371412f054-secret-telemeter-client\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:19.925351 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:19.925333    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvnb2\" (UniqueName: \"kubernetes.io/projected/9030b38c-0472-4cd4-bc5d-65371412f054-kube-api-access-rvnb2\") pod \"telemeter-client-7d8b9bb886-jz9dn\" (UID: \"9030b38c-0472-4cd4-bc5d-65371412f054\") " pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:20.001373 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:20.001288    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"
Sep 17 18:42:20.286084 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:20.286000    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn"]
Sep 17 18:42:20.289616 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:42:20.289588    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9030b38c_0472_4cd4_bc5d_65371412f054.slice/crio-a8657b1f831c3b867f39504fc486a96514647f6e38e3cdd19af184c4bb4d5b43 WatchSource:0}: Error finding container a8657b1f831c3b867f39504fc486a96514647f6e38e3cdd19af184c4bb4d5b43: Status 404 returned error can't find the container with id a8657b1f831c3b867f39504fc486a96514647f6e38e3cdd19af184c4bb4d5b43
Sep 17 18:42:21.118001 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:21.117963    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn" event={"ID":"9030b38c-0472-4cd4-bc5d-65371412f054","Type":"ContainerStarted","Data":"a8657b1f831c3b867f39504fc486a96514647f6e38e3cdd19af184c4bb4d5b43"}
Sep 17 18:42:23.127220 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:23.127173    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn" event={"ID":"9030b38c-0472-4cd4-bc5d-65371412f054","Type":"ContainerStarted","Data":"b2a3c16cd24ee0dde9e5bc1855e4b697fc0be8309841a995d03596295d60d2e7"}
Sep 17 18:42:24.132477 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.132446    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn" event={"ID":"9030b38c-0472-4cd4-bc5d-65371412f054","Type":"ContainerStarted","Data":"17d6fa7dc1334cc2411b89ec0f396f9e473e03b71647d7dcec8b563ec5fdfc51"}
Sep 17 18:42:24.132970 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.132504    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn" event={"ID":"9030b38c-0472-4cd4-bc5d-65371412f054","Type":"ContainerStarted","Data":"9279d3c73f5bedcc2e580824361971b05215460fac94d688df046658fc145bf6"}
Sep 17 18:42:24.157006 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.156935    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7d8b9bb886-jz9dn" podStartSLOduration=2.2718992399999998 podStartE2EDuration="5.156917573s" totalImagesPullingTime="2.885018333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:24.156307626 +0000 UTC m=+165.016132287" watchObservedRunningTime="2026-09-17 18:42:24.156917573 +0000 UTC m=+165.016742222"
Sep 17 18:42:24.245648 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.245615    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d57b77b45-dgnth"]
Sep 17 18:42:24.247804 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.247789    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.261383 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.261359    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d57b77b45-dgnth"]
Sep 17 18:42:24.351056 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.351016    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-trusted-ca-bundle\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.351056 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.351066    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-oauth-config\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.351311 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.351097    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwbj\" (UniqueName: \"kubernetes.io/projected/aee764d6-82f1-4930-ad03-3dca1274661a-kube-api-access-xkwbj\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.351311 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.351164    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-serving-cert\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.351311 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.351206    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-service-ca\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.351311 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.351241    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-console-config\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.351311 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.351257    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-oauth-serving-cert\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.452213 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.452118    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwbj\" (UniqueName: \"kubernetes.io/projected/aee764d6-82f1-4930-ad03-3dca1274661a-kube-api-access-xkwbj\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.452213 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.452167    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-serving-cert\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.452213 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.452195    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-service-ca\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.452492 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.452235    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-console-config\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.452492 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.452357    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-oauth-serving-cert\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.452492 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.452406    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-trusted-ca-bundle\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.452492 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.452452    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-oauth-config\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.452987 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.452956    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-console-config\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.453147 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.453123    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-oauth-serving-cert\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.453237 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.453216    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-service-ca\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.453272 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.453243    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-trusted-ca-bundle\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.454884 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.454863    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-oauth-config\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.455089 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.455069    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-serving-cert\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.461731 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.461712    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwbj\" (UniqueName: \"kubernetes.io/projected/aee764d6-82f1-4930-ad03-3dca1274661a-kube-api-access-xkwbj\") pod \"console-7d57b77b45-dgnth\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") " pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.557058 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.557014    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:24.835147 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:24.835011    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d57b77b45-dgnth"]
Sep 17 18:42:24.837639 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:42:24.837613    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee764d6_82f1_4930_ad03_3dca1274661a.slice/crio-c99067fd0dcadefa05eeea063d1c7a04a79e8580646be150dad5a5eb4635546b WatchSource:0}: Error finding container c99067fd0dcadefa05eeea063d1c7a04a79e8580646be150dad5a5eb4635546b: Status 404 returned error can't find the container with id c99067fd0dcadefa05eeea063d1c7a04a79e8580646be150dad5a5eb4635546b
Sep 17 18:42:25.137783 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:25.137688    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d57b77b45-dgnth" event={"ID":"aee764d6-82f1-4930-ad03-3dca1274661a","Type":"ContainerStarted","Data":"fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b"}
Sep 17 18:42:25.138256 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:25.138097    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d57b77b45-dgnth" event={"ID":"aee764d6-82f1-4930-ad03-3dca1274661a","Type":"ContainerStarted","Data":"c99067fd0dcadefa05eeea063d1c7a04a79e8580646be150dad5a5eb4635546b"}
Sep 17 18:42:25.160516 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:25.160461    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d57b77b45-dgnth" podStartSLOduration=1.160443176 podStartE2EDuration="1.160443176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:25.159640712 +0000 UTC m=+166.019465380" watchObservedRunningTime="2026-09-17 18:42:25.160443176 +0000 UTC m=+166.020267824"
Sep 17 18:42:34.557288 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:34.557252    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:34.557655 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:34.557326    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:34.561991 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:34.561968    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:35.177773 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.177745    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:42:35.247343 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.247306    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fd7cf5f7b-55wgr"]
Sep 17 18:42:35.371538 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.371501    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d899b4755-c2kjf"]
Sep 17 18:42:35.373879 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.373858    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.385781 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.385758    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d899b4755-c2kjf"]
Sep 17 18:42:35.541540 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.541438    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-oauth-serving-cert\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.541540 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.541484    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-console-config\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.541540 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.541515    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-trusted-ca-bundle\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.541822 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.541589    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-serving-cert\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.541822 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.541633    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-service-ca\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.541822 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.541731    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzj9l\" (UniqueName: \"kubernetes.io/projected/4d8d5258-b30f-4796-9e50-da80e93f132b-kube-api-access-gzj9l\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.541822 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.541788    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-oauth-config\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.642965 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.642926    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-serving-cert\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.642965 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.642964    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-service-ca\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.643404 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.643007    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzj9l\" (UniqueName: \"kubernetes.io/projected/4d8d5258-b30f-4796-9e50-da80e93f132b-kube-api-access-gzj9l\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.643404 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.643052    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-oauth-config\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.643404 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.643093    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-oauth-serving-cert\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.643404 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.643118    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-console-config\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.643404 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.643141    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-trusted-ca-bundle\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.643804 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.643779    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-oauth-serving-cert\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.643804 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.643793    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-service-ca\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.643959 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.643868    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-console-config\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.644048 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.644032    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-trusted-ca-bundle\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.646125 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.646101    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-oauth-config\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.646251 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.646234    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-serving-cert\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.651792 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.651773    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzj9l\" (UniqueName: \"kubernetes.io/projected/4d8d5258-b30f-4796-9e50-da80e93f132b-kube-api-access-gzj9l\") pod \"console-d899b4755-c2kjf\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") " pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.682741 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.682701    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:35.955633 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:35.955601    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d899b4755-c2kjf"]
Sep 17 18:42:35.959062 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:42:35.959029    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8d5258_b30f_4796_9e50_da80e93f132b.slice/crio-43c18e3e0172be7f2ec935a7c6f4b11df5a61e762919cc345d6b19f114b8ac86 WatchSource:0}: Error finding container 43c18e3e0172be7f2ec935a7c6f4b11df5a61e762919cc345d6b19f114b8ac86: Status 404 returned error can't find the container with id 43c18e3e0172be7f2ec935a7c6f4b11df5a61e762919cc345d6b19f114b8ac86
Sep 17 18:42:36.179501 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:36.179461    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d899b4755-c2kjf" event={"ID":"4d8d5258-b30f-4796-9e50-da80e93f132b","Type":"ContainerStarted","Data":"0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e"}
Sep 17 18:42:36.179667 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:36.179520    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d899b4755-c2kjf" event={"ID":"4d8d5258-b30f-4796-9e50-da80e93f132b","Type":"ContainerStarted","Data":"43c18e3e0172be7f2ec935a7c6f4b11df5a61e762919cc345d6b19f114b8ac86"}
Sep 17 18:42:36.199408 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:36.199355    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d899b4755-c2kjf" podStartSLOduration=1.199338917 podStartE2EDuration="1.199338917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:36.199062966 +0000 UTC m=+177.058887641" watchObservedRunningTime="2026-09-17 18:42:36.199338917 +0000 UTC m=+177.059163564"
Sep 17 18:42:45.682950 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:45.682901    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:45.683461 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:45.682975    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:45.687563 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:45.687541    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:46.220460 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:46.220430    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:42:46.270489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:42:46.270449    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d57b77b45-dgnth"]
Sep 17 18:43:00.272245 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.272185    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fd7cf5f7b-55wgr" podUID="e34a19e1-5079-4b21-90d4-61977a8597c4" containerName="console" containerID="cri-o://32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d" gracePeriod=15
Sep 17 18:43:00.515966 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.515942    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:43:00.537073 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.536995    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-service-ca\") pod \"e34a19e1-5079-4b21-90d4-61977a8597c4\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") "
Sep 17 18:43:00.537073 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.537028    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-console-config\") pod \"e34a19e1-5079-4b21-90d4-61977a8597c4\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") "
Sep 17 18:43:00.537073 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.537055    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-oauth-config\") pod \"e34a19e1-5079-4b21-90d4-61977a8597c4\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") "
Sep 17 18:43:00.537308 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.537095    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-trusted-ca-bundle\") pod \"e34a19e1-5079-4b21-90d4-61977a8597c4\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") "
Sep 17 18:43:00.537308 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.537124    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwhjn\" (UniqueName: \"kubernetes.io/projected/e34a19e1-5079-4b21-90d4-61977a8597c4-kube-api-access-hwhjn\") pod \"e34a19e1-5079-4b21-90d4-61977a8597c4\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") "
Sep 17 18:43:00.537308 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.537156    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-oauth-serving-cert\") pod \"e34a19e1-5079-4b21-90d4-61977a8597c4\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") "
Sep 17 18:43:00.537308 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.537200    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-serving-cert\") pod \"e34a19e1-5079-4b21-90d4-61977a8597c4\" (UID: \"e34a19e1-5079-4b21-90d4-61977a8597c4\") "
Sep 17 18:43:00.537499 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.537467    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-console-config" (OuterVolumeSpecName: "console-config") pod "e34a19e1-5079-4b21-90d4-61977a8597c4" (UID: "e34a19e1-5079-4b21-90d4-61977a8597c4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:00.537559 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.537446    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-service-ca" (OuterVolumeSpecName: "service-ca") pod "e34a19e1-5079-4b21-90d4-61977a8597c4" (UID: "e34a19e1-5079-4b21-90d4-61977a8597c4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:00.537628 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.537602    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e34a19e1-5079-4b21-90d4-61977a8597c4" (UID: "e34a19e1-5079-4b21-90d4-61977a8597c4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:00.537674 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.537633    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e34a19e1-5079-4b21-90d4-61977a8597c4" (UID: "e34a19e1-5079-4b21-90d4-61977a8597c4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:00.539888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.539828    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34a19e1-5079-4b21-90d4-61977a8597c4-kube-api-access-hwhjn" (OuterVolumeSpecName: "kube-api-access-hwhjn") pod "e34a19e1-5079-4b21-90d4-61977a8597c4" (UID: "e34a19e1-5079-4b21-90d4-61977a8597c4"). InnerVolumeSpecName "kube-api-access-hwhjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:00.540213 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.540189    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e34a19e1-5079-4b21-90d4-61977a8597c4" (UID: "e34a19e1-5079-4b21-90d4-61977a8597c4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:00.540428 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.540403    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e34a19e1-5079-4b21-90d4-61977a8597c4" (UID: "e34a19e1-5079-4b21-90d4-61977a8597c4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:00.638677 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.638635    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.638677 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.638667    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-console-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.638677 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.638676    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-oauth-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.638677 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.638684    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-trusted-ca-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.638966 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.638692    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwhjn\" (UniqueName: \"kubernetes.io/projected/e34a19e1-5079-4b21-90d4-61977a8597c4-kube-api-access-hwhjn\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.638966 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.638702    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e34a19e1-5079-4b21-90d4-61977a8597c4-oauth-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.638966 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:00.638710    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e34a19e1-5079-4b21-90d4-61977a8597c4-console-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:01.271685 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:01.271650    2586 generic.go:356] "Generic (PLEG): container finished" podID="e34a19e1-5079-4b21-90d4-61977a8597c4" containerID="32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d" exitCode=0
Sep 17 18:43:01.271884 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:01.271716    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd7cf5f7b-55wgr"
Sep 17 18:43:01.271884 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:01.271734    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd7cf5f7b-55wgr" event={"ID":"e34a19e1-5079-4b21-90d4-61977a8597c4","Type":"ContainerDied","Data":"32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d"}
Sep 17 18:43:01.271884 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:01.271815    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd7cf5f7b-55wgr" event={"ID":"e34a19e1-5079-4b21-90d4-61977a8597c4","Type":"ContainerDied","Data":"cb3966aa113d6ce534ef404e51ff845b0283f0805d90ae6f003485c3caeb256e"}
Sep 17 18:43:01.271884 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:01.271850    2586 scope.go:117] "RemoveContainer" containerID="32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d"
Sep 17 18:43:01.280632 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:01.280490    2586 scope.go:117] "RemoveContainer" containerID="32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d"
Sep 17 18:43:01.280896 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:43:01.280771    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d\": container with ID starting with 32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d not found: ID does not exist" containerID="32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d"
Sep 17 18:43:01.280896 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:01.280796    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d"} err="failed to get container status \"32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d\": rpc error: code = NotFound desc = could not find container \"32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d\": container with ID starting with 32b8744374bc105a49baf011f8348d320364ad1442bef7b7c13b0579c3da656d not found: ID does not exist"
Sep 17 18:43:01.299042 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:01.298993    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fd7cf5f7b-55wgr"]
Sep 17 18:43:01.314410 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:01.314384    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fd7cf5f7b-55wgr"]
Sep 17 18:43:01.456870 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:01.456818    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e34a19e1-5079-4b21-90d4-61977a8597c4" path="/var/lib/kubelet/pods/e34a19e1-5079-4b21-90d4-61977a8597c4/volumes"
Sep 17 18:43:11.290582 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.290536    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d57b77b45-dgnth" podUID="aee764d6-82f1-4930-ad03-3dca1274661a" containerName="console" containerID="cri-o://fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b" gracePeriod=15
Sep 17 18:43:11.543657 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.543597    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:43:11.619802 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.619758    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-oauth-config\") pod \"aee764d6-82f1-4930-ad03-3dca1274661a\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") "
Sep 17 18:43:11.619802 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.619809    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-trusted-ca-bundle\") pod \"aee764d6-82f1-4930-ad03-3dca1274661a\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") "
Sep 17 18:43:11.620074 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.619887    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkwbj\" (UniqueName: \"kubernetes.io/projected/aee764d6-82f1-4930-ad03-3dca1274661a-kube-api-access-xkwbj\") pod \"aee764d6-82f1-4930-ad03-3dca1274661a\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") "
Sep 17 18:43:11.620074 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.619909    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-console-config\") pod \"aee764d6-82f1-4930-ad03-3dca1274661a\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") "
Sep 17 18:43:11.620074 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.619938    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-serving-cert\") pod \"aee764d6-82f1-4930-ad03-3dca1274661a\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") "
Sep 17 18:43:11.620074 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.619967    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-service-ca\") pod \"aee764d6-82f1-4930-ad03-3dca1274661a\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") "
Sep 17 18:43:11.620074 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.620006    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-oauth-serving-cert\") pod \"aee764d6-82f1-4930-ad03-3dca1274661a\" (UID: \"aee764d6-82f1-4930-ad03-3dca1274661a\") "
Sep 17 18:43:11.620411 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.620348    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-console-config" (OuterVolumeSpecName: "console-config") pod "aee764d6-82f1-4930-ad03-3dca1274661a" (UID: "aee764d6-82f1-4930-ad03-3dca1274661a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:11.620411 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.620383    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-service-ca" (OuterVolumeSpecName: "service-ca") pod "aee764d6-82f1-4930-ad03-3dca1274661a" (UID: "aee764d6-82f1-4930-ad03-3dca1274661a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:11.620569 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.620460    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aee764d6-82f1-4930-ad03-3dca1274661a" (UID: "aee764d6-82f1-4930-ad03-3dca1274661a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:11.620569 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.620500    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aee764d6-82f1-4930-ad03-3dca1274661a" (UID: "aee764d6-82f1-4930-ad03-3dca1274661a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:11.622207 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.622187    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee764d6-82f1-4930-ad03-3dca1274661a-kube-api-access-xkwbj" (OuterVolumeSpecName: "kube-api-access-xkwbj") pod "aee764d6-82f1-4930-ad03-3dca1274661a" (UID: "aee764d6-82f1-4930-ad03-3dca1274661a"). InnerVolumeSpecName "kube-api-access-xkwbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:11.622702 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.622678    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aee764d6-82f1-4930-ad03-3dca1274661a" (UID: "aee764d6-82f1-4930-ad03-3dca1274661a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:11.622788 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.622697    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aee764d6-82f1-4930-ad03-3dca1274661a" (UID: "aee764d6-82f1-4930-ad03-3dca1274661a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:11.720547 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.720518    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:11.720547 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.720544    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-oauth-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:11.720547 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.720554    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-oauth-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:11.720892 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.720562    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-trusted-ca-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:11.720892 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.720570    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkwbj\" (UniqueName: \"kubernetes.io/projected/aee764d6-82f1-4930-ad03-3dca1274661a-kube-api-access-xkwbj\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:11.720892 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.720578    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aee764d6-82f1-4930-ad03-3dca1274661a-console-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:11.720892 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:11.720587    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aee764d6-82f1-4930-ad03-3dca1274661a-console-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:43:12.313470 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:12.313426    2586 generic.go:356] "Generic (PLEG): container finished" podID="aee764d6-82f1-4930-ad03-3dca1274661a" containerID="fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b" exitCode=0
Sep 17 18:43:12.313470 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:12.313470    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d57b77b45-dgnth"
Sep 17 18:43:12.313961 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:12.313490    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d57b77b45-dgnth" event={"ID":"aee764d6-82f1-4930-ad03-3dca1274661a","Type":"ContainerDied","Data":"fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b"}
Sep 17 18:43:12.313961 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:12.313536    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d57b77b45-dgnth" event={"ID":"aee764d6-82f1-4930-ad03-3dca1274661a","Type":"ContainerDied","Data":"c99067fd0dcadefa05eeea063d1c7a04a79e8580646be150dad5a5eb4635546b"}
Sep 17 18:43:12.313961 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:12.313558    2586 scope.go:117] "RemoveContainer" containerID="fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b"
Sep 17 18:43:12.322149 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:12.322129    2586 scope.go:117] "RemoveContainer" containerID="fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b"
Sep 17 18:43:12.322419 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:43:12.322401    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b\": container with ID starting with fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b not found: ID does not exist" containerID="fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b"
Sep 17 18:43:12.322467 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:12.322427    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b"} err="failed to get container status \"fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b\": rpc error: code = NotFound desc = could not find container \"fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b\": container with ID starting with fe52a086d1d69cd50216ab35b2c2711e70b08fcdccf27c413c91bd9fdeaf4f0b not found: ID does not exist"
Sep 17 18:43:12.337134 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:12.337109    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d57b77b45-dgnth"]
Sep 17 18:43:12.341142 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:12.341122    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d57b77b45-dgnth"]
Sep 17 18:43:13.457489 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:43:13.457454    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee764d6-82f1-4930-ad03-3dca1274661a" path="/var/lib/kubelet/pods/aee764d6-82f1-4930-ad03-3dca1274661a/volumes"
Sep 17 18:44:21.539522 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.539487    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"]
Sep 17 18:44:21.539960 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.539777    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee764d6-82f1-4930-ad03-3dca1274661a" containerName="console"
Sep 17 18:44:21.539960 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.539792    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee764d6-82f1-4930-ad03-3dca1274661a" containerName="console"
Sep 17 18:44:21.539960 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.539810    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e34a19e1-5079-4b21-90d4-61977a8597c4" containerName="console"
Sep 17 18:44:21.539960 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.539816    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e34a19e1-5079-4b21-90d4-61977a8597c4" containerName="console"
Sep 17 18:44:21.539960 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.539884    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e34a19e1-5079-4b21-90d4-61977a8597c4" containerName="console"
Sep 17 18:44:21.539960 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.539895    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee764d6-82f1-4930-ad03-3dca1274661a" containerName="console"
Sep 17 18:44:21.542785 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.542765    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:21.545250 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.545228    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xl97j\""
Sep 17 18:44:21.545514 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.545500    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:44:21.546319 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.546306    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:44:21.553606 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.553585    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"]
Sep 17 18:44:21.635680 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.635645    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrf5k\" (UniqueName: \"kubernetes.io/projected/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-kube-api-access-nrf5k\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:21.635878 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.635755    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:21.635878 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.635783    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:21.736597 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.736564    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:21.736597 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.736600    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:21.736811 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.736644    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrf5k\" (UniqueName: \"kubernetes.io/projected/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-kube-api-access-nrf5k\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:21.737063 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.737044    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:21.737116 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.737058    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:21.747056 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.747030    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrf5k\" (UniqueName: \"kubernetes.io/projected/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-kube-api-access-nrf5k\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:21.851860 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:21.851803    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:22.123540 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:22.123453    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"]
Sep 17 18:44:22.126600 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:44:22.126577    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52cd2fb_2b26_4add_bafa_1c1c1ab2d5e7.slice/crio-f2dad2a6d1e13701c6a879bf313204c5a80ad9cd7feb00df62c46f9bcc549596 WatchSource:0}: Error finding container f2dad2a6d1e13701c6a879bf313204c5a80ad9cd7feb00df62c46f9bcc549596: Status 404 returned error can't find the container with id f2dad2a6d1e13701c6a879bf313204c5a80ad9cd7feb00df62c46f9bcc549596
Sep 17 18:44:22.569927 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:22.569896    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882" event={"ID":"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7","Type":"ContainerStarted","Data":"f2dad2a6d1e13701c6a879bf313204c5a80ad9cd7feb00df62c46f9bcc549596"}
Sep 17 18:44:29.598375 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:29.598336    2586 generic.go:356] "Generic (PLEG): container finished" podID="c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" containerID="072b3062b52876ab9a3e0d0307e7121708745dc1fbb792d38cd27ad32ea894e1" exitCode=0
Sep 17 18:44:29.598752 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:29.598396    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882" event={"ID":"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7","Type":"ContainerDied","Data":"072b3062b52876ab9a3e0d0307e7121708745dc1fbb792d38cd27ad32ea894e1"}
Sep 17 18:44:32.612625 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:32.612583    2586 generic.go:356] "Generic (PLEG): container finished" podID="c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" containerID="d95f86555d87618c75ebc2934b8d54e1d4a376b0f717f4666a4395eaec8bf07c" exitCode=0
Sep 17 18:44:32.613099 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:32.612662    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882" event={"ID":"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7","Type":"ContainerDied","Data":"d95f86555d87618c75ebc2934b8d54e1d4a376b0f717f4666a4395eaec8bf07c"}
Sep 17 18:44:39.644111 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:39.644068    2586 generic.go:356] "Generic (PLEG): container finished" podID="c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" containerID="13d615d13bdafce59ee13748eb1e374060a75faa11a2df17e21559c8aa366bcb" exitCode=0
Sep 17 18:44:39.644477 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:39.644143    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882" event={"ID":"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7","Type":"ContainerDied","Data":"13d615d13bdafce59ee13748eb1e374060a75faa11a2df17e21559c8aa366bcb"}
Sep 17 18:44:40.771927 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:40.771901    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:40.890446 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:40.890417    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrf5k\" (UniqueName: \"kubernetes.io/projected/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-kube-api-access-nrf5k\") pod \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") "
Sep 17 18:44:40.890634 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:40.890466    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-bundle\") pod \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") "
Sep 17 18:44:40.890634 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:40.890487    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-util\") pod \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\" (UID: \"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7\") "
Sep 17 18:44:40.891267 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:40.891239    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-bundle" (OuterVolumeSpecName: "bundle") pod "c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" (UID: "c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:40.892775 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:40.892751    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-kube-api-access-nrf5k" (OuterVolumeSpecName: "kube-api-access-nrf5k") pod "c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" (UID: "c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7"). InnerVolumeSpecName "kube-api-access-nrf5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:40.897781 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:40.897753    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-util" (OuterVolumeSpecName: "util") pod "c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" (UID: "c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:40.991314 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:40.991268    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrf5k\" (UniqueName: \"kubernetes.io/projected/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-kube-api-access-nrf5k\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:44:40.991314 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:40.991310    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:44:40.991314 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:40.991321    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7-util\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:44:41.302875 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:41.302781    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:44:41.306824 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:41.306801    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:44:41.310231 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:41.310213    2586 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:44:41.653085 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:41.653050    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882" event={"ID":"c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7","Type":"ContainerDied","Data":"f2dad2a6d1e13701c6a879bf313204c5a80ad9cd7feb00df62c46f9bcc549596"}
Sep 17 18:44:41.653281 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:41.653088    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aqx882"
Sep 17 18:44:41.653281 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:41.653095    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2dad2a6d1e13701c6a879bf313204c5a80ad9cd7feb00df62c46f9bcc549596"
Sep 17 18:44:44.164982 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.164949    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"]
Sep 17 18:44:44.165363 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.165215    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" containerName="pull"
Sep 17 18:44:44.165363 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.165225    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" containerName="pull"
Sep 17 18:44:44.165363 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.165237    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" containerName="extract"
Sep 17 18:44:44.165363 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.165242    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" containerName="extract"
Sep 17 18:44:44.165363 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.165250    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" containerName="util"
Sep 17 18:44:44.165363 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.165256    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" containerName="util"
Sep 17 18:44:44.165363 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.165297    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52cd2fb-2b26-4add-bafa-1c1c1ab2d5e7" containerName="extract"
Sep 17 18:44:44.210855 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.210803    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"]
Sep 17 18:44:44.211017 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.210929    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"
Sep 17 18:44:44.213935 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.213913    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-vwbb8\""
Sep 17 18:44:44.214074 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.213915    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 18:44:44.214074 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.213925    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 18:44:44.214189 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.214144    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 18:44:44.316768 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.316730    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfg75\" (UniqueName: \"kubernetes.io/projected/1612b410-5074-4baa-995f-a6edd78f2c01-kube-api-access-hfg75\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rhvq9\" (UID: \"1612b410-5074-4baa-995f-a6edd78f2c01\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"
Sep 17 18:44:44.316953 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.316777    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1612b410-5074-4baa-995f-a6edd78f2c01-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rhvq9\" (UID: \"1612b410-5074-4baa-995f-a6edd78f2c01\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"
Sep 17 18:44:44.418065 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.417970    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1612b410-5074-4baa-995f-a6edd78f2c01-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rhvq9\" (UID: \"1612b410-5074-4baa-995f-a6edd78f2c01\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"
Sep 17 18:44:44.418065 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.418069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfg75\" (UniqueName: \"kubernetes.io/projected/1612b410-5074-4baa-995f-a6edd78f2c01-kube-api-access-hfg75\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rhvq9\" (UID: \"1612b410-5074-4baa-995f-a6edd78f2c01\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"
Sep 17 18:44:44.420590 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.420568    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1612b410-5074-4baa-995f-a6edd78f2c01-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rhvq9\" (UID: \"1612b410-5074-4baa-995f-a6edd78f2c01\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"
Sep 17 18:44:44.427680 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.427640    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfg75\" (UniqueName: \"kubernetes.io/projected/1612b410-5074-4baa-995f-a6edd78f2c01-kube-api-access-hfg75\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rhvq9\" (UID: \"1612b410-5074-4baa-995f-a6edd78f2c01\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"
Sep 17 18:44:44.520992 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.520957    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"
Sep 17 18:44:44.815510 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.815481    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"]
Sep 17 18:44:44.819226 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:44:44.819197    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1612b410_5074_4baa_995f_a6edd78f2c01.slice/crio-fc653d5a6b1c9706f8700185f24aa58e5f1e14d606d5f5b63835afd101fe2365 WatchSource:0}: Error finding container fc653d5a6b1c9706f8700185f24aa58e5f1e14d606d5f5b63835afd101fe2365: Status 404 returned error can't find the container with id fc653d5a6b1c9706f8700185f24aa58e5f1e14d606d5f5b63835afd101fe2365
Sep 17 18:44:44.821319 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:44.821300    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:44:45.669080 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:45.669040    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9" event={"ID":"1612b410-5074-4baa-995f-a6edd78f2c01","Type":"ContainerStarted","Data":"fc653d5a6b1c9706f8700185f24aa58e5f1e14d606d5f5b63835afd101fe2365"}
Sep 17 18:44:48.688782 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.688750    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9" event={"ID":"1612b410-5074-4baa-995f-a6edd78f2c01","Type":"ContainerStarted","Data":"1d9f760483071c72861faa49d514ba982f9d77aa217163bd108c2b1c4b2cc118"}
Sep 17 18:44:48.689216 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.688890    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"
Sep 17 18:44:48.712453 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.712404    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9" podStartSLOduration=1.370975064 podStartE2EDuration="4.712389228s" totalImagesPullingTime="3.341414164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:48.711662767 +0000 UTC m=+309.571487434" watchObservedRunningTime="2026-09-17 18:44:48.712389228 +0000 UTC m=+309.572213876"
Sep 17 18:44:48.884122 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.884090    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-mgzqb"]
Sep 17 18:44:48.886221 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.886204    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:48.888649 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.888627    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 18:44:48.888770 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.888721    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 18:44:48.888770 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.888747    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vltbf\""
Sep 17 18:44:48.895673 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.895649    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-mgzqb"]
Sep 17 18:44:48.954415 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.954329    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/44a7272e-6054-4963-af0b-95b442de83b8-cabundle0\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:48.954415 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.954369    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncsmg\" (UniqueName: \"kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-kube-api-access-ncsmg\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:48.954596 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:48.954475    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-certificates\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:49.055608 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.055576    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-certificates\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:49.055799 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.055629    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/44a7272e-6054-4963-af0b-95b442de83b8-cabundle0\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:49.055799 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.055658    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncsmg\" (UniqueName: \"kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-kube-api-access-ncsmg\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:49.055799 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.055732    2586 secret.go:281] references non-existent secret key: ca.crt
Sep 17 18:44:49.055799 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.055755    2586 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 18:44:49.055799 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.055764    2586 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-mgzqb: references non-existent secret key: ca.crt
Sep 17 18:44:49.056139 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.055849    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-certificates podName:44a7272e-6054-4963-af0b-95b442de83b8 nodeName:}" failed. No retries permitted until 2026-09-17 18:44:49.555814209 +0000 UTC m=+310.415638853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-certificates") pod "keda-operator-579c8958f7-mgzqb" (UID: "44a7272e-6054-4963-af0b-95b442de83b8") : references non-existent secret key: ca.crt
Sep 17 18:44:49.056270 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.056249    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/44a7272e-6054-4963-af0b-95b442de83b8-cabundle0\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:49.064631 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.064610    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncsmg\" (UniqueName: \"kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-kube-api-access-ncsmg\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:49.081402 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.081376    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"]
Sep 17 18:44:49.083946 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.083929    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.086253 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.086233    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 18:44:49.097359 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.097338    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"]
Sep 17 18:44:49.156219 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.156187    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-tv667\" (UID: \"5bf19dd0-c2a1-44a6-a864-b88fc73a040a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.156421 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.156233    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tv667\" (UID: \"5bf19dd0-c2a1-44a6-a864-b88fc73a040a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.156421 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.156352    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9zr\" (UniqueName: \"kubernetes.io/projected/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-kube-api-access-ns9zr\") pod \"keda-metrics-apiserver-77857b7b98-tv667\" (UID: \"5bf19dd0-c2a1-44a6-a864-b88fc73a040a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.256961 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.256865    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-tv667\" (UID: \"5bf19dd0-c2a1-44a6-a864-b88fc73a040a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.256961 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.256913    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tv667\" (UID: \"5bf19dd0-c2a1-44a6-a864-b88fc73a040a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.257187 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.256993    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9zr\" (UniqueName: \"kubernetes.io/projected/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-kube-api-access-ns9zr\") pod \"keda-metrics-apiserver-77857b7b98-tv667\" (UID: \"5bf19dd0-c2a1-44a6-a864-b88fc73a040a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.257187 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.257083    2586 secret.go:281] references non-existent secret key: tls.crt
Sep 17 18:44:49.257187 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.257105    2586 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 18:44:49.257187 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.257122    2586 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 17 18:44:49.257187 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.257139    2586 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-tv667: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 17 18:44:49.257187 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.257190    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-certificates podName:5bf19dd0-c2a1-44a6-a864-b88fc73a040a nodeName:}" failed. No retries permitted until 2026-09-17 18:44:49.757175307 +0000 UTC m=+310.616999933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-certificates") pod "keda-metrics-apiserver-77857b7b98-tv667" (UID: "5bf19dd0-c2a1-44a6-a864-b88fc73a040a") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 17 18:44:49.257455 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.257251    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-tv667\" (UID: \"5bf19dd0-c2a1-44a6-a864-b88fc73a040a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.268164 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.268130    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9zr\" (UniqueName: \"kubernetes.io/projected/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-kube-api-access-ns9zr\") pod \"keda-metrics-apiserver-77857b7b98-tv667\" (UID: \"5bf19dd0-c2a1-44a6-a864-b88fc73a040a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.563087 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.562969    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-certificates\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:49.563352 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.563283    2586 secret.go:281] references non-existent secret key: ca.crt
Sep 17 18:44:49.563352 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.563304    2586 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 18:44:49.563352 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.563316    2586 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-mgzqb: references non-existent secret key: ca.crt
Sep 17 18:44:49.563460 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.563366    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-certificates podName:44a7272e-6054-4963-af0b-95b442de83b8 nodeName:}" failed. No retries permitted until 2026-09-17 18:44:50.563352699 +0000 UTC m=+311.423177324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-certificates") pod "keda-operator-579c8958f7-mgzqb" (UID: "44a7272e-6054-4963-af0b-95b442de83b8") : references non-existent secret key: ca.crt
Sep 17 18:44:49.749862 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.749811    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-4l9pz"]
Sep 17 18:44:49.752468 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.752445    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:44:49.755027 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.755006    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 18:44:49.765266 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.765241    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tv667\" (UID: \"5bf19dd0-c2a1-44a6-a864-b88fc73a040a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.768302 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.768278    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5bf19dd0-c2a1-44a6-a864-b88fc73a040a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tv667\" (UID: \"5bf19dd0-c2a1-44a6-a864-b88fc73a040a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:49.768536 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.768512    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-4l9pz"]
Sep 17 18:44:49.866757 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.866720    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d29d56a6-ea6e-44d5-a296-312a1b4bce57-certificates\") pod \"keda-admission-6c4d864b86-4l9pz\" (UID: \"d29d56a6-ea6e-44d5-a296-312a1b4bce57\") " pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:44:49.866947 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.866767    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8m2r\" (UniqueName: \"kubernetes.io/projected/d29d56a6-ea6e-44d5-a296-312a1b4bce57-kube-api-access-s8m2r\") pod \"keda-admission-6c4d864b86-4l9pz\" (UID: \"d29d56a6-ea6e-44d5-a296-312a1b4bce57\") " pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:44:49.967923 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.967883    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d29d56a6-ea6e-44d5-a296-312a1b4bce57-certificates\") pod \"keda-admission-6c4d864b86-4l9pz\" (UID: \"d29d56a6-ea6e-44d5-a296-312a1b4bce57\") " pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:44:49.968120 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.967952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8m2r\" (UniqueName: \"kubernetes.io/projected/d29d56a6-ea6e-44d5-a296-312a1b4bce57-kube-api-access-s8m2r\") pod \"keda-admission-6c4d864b86-4l9pz\" (UID: \"d29d56a6-ea6e-44d5-a296-312a1b4bce57\") " pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:44:49.968120 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.968051    2586 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 18:44:49.968120 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.968078    2586 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-4l9pz: secret "keda-admission-webhooks-certs" not found
Sep 17 18:44:49.968276 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:44:49.968144    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d29d56a6-ea6e-44d5-a296-312a1b4bce57-certificates podName:d29d56a6-ea6e-44d5-a296-312a1b4bce57 nodeName:}" failed. No retries permitted until 2026-09-17 18:44:50.468126618 +0000 UTC m=+311.327951250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d29d56a6-ea6e-44d5-a296-312a1b4bce57-certificates") pod "keda-admission-6c4d864b86-4l9pz" (UID: "d29d56a6-ea6e-44d5-a296-312a1b4bce57") : secret "keda-admission-webhooks-certs" not found
Sep 17 18:44:49.980534 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.980504    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8m2r\" (UniqueName: \"kubernetes.io/projected/d29d56a6-ea6e-44d5-a296-312a1b4bce57-kube-api-access-s8m2r\") pod \"keda-admission-6c4d864b86-4l9pz\" (UID: \"d29d56a6-ea6e-44d5-a296-312a1b4bce57\") " pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:44:49.996236 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:49.996207    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:50.271049 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:50.271022    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"]
Sep 17 18:44:50.273767 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:44:50.273742    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf19dd0_c2a1_44a6_a864_b88fc73a040a.slice/crio-195455dcaaf36fe8bc3c52064acea2f1d4ede0e56ec53ad21ed042698f35797c WatchSource:0}: Error finding container 195455dcaaf36fe8bc3c52064acea2f1d4ede0e56ec53ad21ed042698f35797c: Status 404 returned error can't find the container with id 195455dcaaf36fe8bc3c52064acea2f1d4ede0e56ec53ad21ed042698f35797c
Sep 17 18:44:50.472906 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:50.472862    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d29d56a6-ea6e-44d5-a296-312a1b4bce57-certificates\") pod \"keda-admission-6c4d864b86-4l9pz\" (UID: \"d29d56a6-ea6e-44d5-a296-312a1b4bce57\") " pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:44:50.475366 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:50.475333    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d29d56a6-ea6e-44d5-a296-312a1b4bce57-certificates\") pod \"keda-admission-6c4d864b86-4l9pz\" (UID: \"d29d56a6-ea6e-44d5-a296-312a1b4bce57\") " pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:44:50.573590 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:50.573557    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-certificates\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:50.576098 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:50.576072    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/44a7272e-6054-4963-af0b-95b442de83b8-certificates\") pod \"keda-operator-579c8958f7-mgzqb\" (UID: \"44a7272e-6054-4963-af0b-95b442de83b8\") " pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:50.662722 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:50.662686    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:44:50.696328 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:50.696295    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:50.697430 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:50.697401    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667" event={"ID":"5bf19dd0-c2a1-44a6-a864-b88fc73a040a","Type":"ContainerStarted","Data":"195455dcaaf36fe8bc3c52064acea2f1d4ede0e56ec53ad21ed042698f35797c"}
Sep 17 18:44:50.964065 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:50.963978    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-4l9pz"]
Sep 17 18:44:50.966292 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:44:50.966263    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29d56a6_ea6e_44d5_a296_312a1b4bce57.slice/crio-808f9b2f61cbc788d28764d2cff7ac3ab3fda4cb1e9b6232ad789217524299d1 WatchSource:0}: Error finding container 808f9b2f61cbc788d28764d2cff7ac3ab3fda4cb1e9b6232ad789217524299d1: Status 404 returned error can't find the container with id 808f9b2f61cbc788d28764d2cff7ac3ab3fda4cb1e9b6232ad789217524299d1
Sep 17 18:44:51.012664 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:51.012634    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-mgzqb"]
Sep 17 18:44:51.014611 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:44:51.014587    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a7272e_6054_4963_af0b_95b442de83b8.slice/crio-2b1181c341316d075408dfa4a7fc508b278caa5eed30ca9a9fe07518896e5d3f WatchSource:0}: Error finding container 2b1181c341316d075408dfa4a7fc508b278caa5eed30ca9a9fe07518896e5d3f: Status 404 returned error can't find the container with id 2b1181c341316d075408dfa4a7fc508b278caa5eed30ca9a9fe07518896e5d3f
Sep 17 18:44:51.703968 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:51.703926    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-mgzqb" event={"ID":"44a7272e-6054-4963-af0b-95b442de83b8","Type":"ContainerStarted","Data":"2b1181c341316d075408dfa4a7fc508b278caa5eed30ca9a9fe07518896e5d3f"}
Sep 17 18:44:51.705195 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:51.705169    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-4l9pz" event={"ID":"d29d56a6-ea6e-44d5-a296-312a1b4bce57","Type":"ContainerStarted","Data":"808f9b2f61cbc788d28764d2cff7ac3ab3fda4cb1e9b6232ad789217524299d1"}
Sep 17 18:44:53.716299 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:53.716262    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-4l9pz" event={"ID":"d29d56a6-ea6e-44d5-a296-312a1b4bce57","Type":"ContainerStarted","Data":"8d4ffc7ff0f79576f872b9c5c1282b2c7e127900c479ab3fc9f5e29e2c573065"}
Sep 17 18:44:53.716734 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:53.716372    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:44:53.736379 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:53.736324    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-4l9pz" podStartSLOduration=2.907197266 podStartE2EDuration="4.736301533s" totalImagesPullingTime="1.829104267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:53.73387717 +0000 UTC m=+314.593701843" watchObservedRunningTime="2026-09-17 18:44:53.736301533 +0000 UTC m=+314.596126183"
Sep 17 18:44:54.721848 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:54.721793    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667" event={"ID":"5bf19dd0-c2a1-44a6-a864-b88fc73a040a","Type":"ContainerStarted","Data":"f92d37a9c40edc4d8983f4b2a8d6cee98cf78145e92a0666f93c88325c2e1bc3"}
Sep 17 18:44:54.722208 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:54.722118    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:44:54.740929 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:54.740875    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667" podStartSLOduration=1.7595840200000001 podStartE2EDuration="5.740853693s" totalImagesPullingTime="3.981269673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:54.738051753 +0000 UTC m=+315.597876412" watchObservedRunningTime="2026-09-17 18:44:54.740853693 +0000 UTC m=+315.600678344"
Sep 17 18:44:56.733359 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:56.733321    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-mgzqb" event={"ID":"44a7272e-6054-4963-af0b-95b442de83b8","Type":"ContainerStarted","Data":"f24a61f1a6ba7a693aa57798cf0e4d60922c09819b17b24f6d80da22724c15a2"}
Sep 17 18:44:56.733794 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:56.733448    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:44:56.752192 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:44:56.752144    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-mgzqb" podStartSLOduration=3.705666128 podStartE2EDuration="8.752128766s" totalImagesPullingTime="5.046462638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:56.75064716 +0000 UTC m=+317.610471829" watchObservedRunningTime="2026-09-17 18:44:56.752128766 +0000 UTC m=+317.611953414"
Sep 17 18:45:05.731575 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:05.731547    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tv667"
Sep 17 18:45:09.695480 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:09.695448    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rhvq9"
Sep 17 18:45:14.724380 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:14.724343    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-4l9pz"
Sep 17 18:45:17.739684 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:17.739661    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-mgzqb"
Sep 17 18:45:41.961705 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:41.961623    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"]
Sep 17 18:45:41.969290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:41.969265    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:41.972306 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:41.972278    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:45:41.972306 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:41.972301    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xl97j\""
Sep 17 18:45:41.973284 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:41.973264    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:45:41.976531 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:41.976508    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"]
Sep 17 18:45:41.998723 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:41.998686    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:41.998902 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:41.998733    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpjq\" (UniqueName: \"kubernetes.io/projected/d907546c-6c1a-4090-91d5-9b8f23dedd4a-kube-api-access-xlpjq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:41.998902 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:41.998784    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:42.099282 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.099244    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:42.099282 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.099290    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpjq\" (UniqueName: \"kubernetes.io/projected/d907546c-6c1a-4090-91d5-9b8f23dedd4a-kube-api-access-xlpjq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:42.099543 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.099329    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:42.099714 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.099689    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:42.099788 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.099739    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:42.110100 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.110076    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpjq\" (UniqueName: \"kubernetes.io/projected/d907546c-6c1a-4090-91d5-9b8f23dedd4a-kube-api-access-xlpjq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:42.280644 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.280560    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:42.554946 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.554918    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"]
Sep 17 18:45:42.557443 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:45:42.557402    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd907546c_6c1a_4090_91d5_9b8f23dedd4a.slice/crio-88a52d4529f3cdce50bcaac6ff94895c052a7b01c2d21fc11d180b0b09e80b19 WatchSource:0}: Error finding container 88a52d4529f3cdce50bcaac6ff94895c052a7b01c2d21fc11d180b0b09e80b19: Status 404 returned error can't find the container with id 88a52d4529f3cdce50bcaac6ff94895c052a7b01c2d21fc11d180b0b09e80b19
Sep 17 18:45:42.926583 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.926545    2586 generic.go:356] "Generic (PLEG): container finished" podID="d907546c-6c1a-4090-91d5-9b8f23dedd4a" containerID="64a5df4b7ea5bcfee2f542ee4f79c72ce0e5ee96e2964f46efdf16e29b8c1d36" exitCode=0
Sep 17 18:45:42.926766 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.926600    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6" event={"ID":"d907546c-6c1a-4090-91d5-9b8f23dedd4a","Type":"ContainerDied","Data":"64a5df4b7ea5bcfee2f542ee4f79c72ce0e5ee96e2964f46efdf16e29b8c1d36"}
Sep 17 18:45:42.926766 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:42.926683    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6" event={"ID":"d907546c-6c1a-4090-91d5-9b8f23dedd4a","Type":"ContainerStarted","Data":"88a52d4529f3cdce50bcaac6ff94895c052a7b01c2d21fc11d180b0b09e80b19"}
Sep 17 18:45:43.003430 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.003395    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fc976d4b7-m2j4h"]
Sep 17 18:45:43.005900 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.005884    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.021055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.021025    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc976d4b7-m2j4h"]
Sep 17 18:45:43.106623 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.106580    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-console-config\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.106623 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.106633    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-service-ca\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.106933 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.106709    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-oauth-serving-cert\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.106933 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.106746    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e923894-d919-4eaf-a313-0643ba059228-console-serving-cert\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.106933 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.106771    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-trusted-ca-bundle\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.106933 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.106788    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcf84\" (UniqueName: \"kubernetes.io/projected/8e923894-d919-4eaf-a313-0643ba059228-kube-api-access-kcf84\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.106933 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.106863    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e923894-d919-4eaf-a313-0643ba059228-console-oauth-config\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.207573 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.207484    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-console-config\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.207573 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.207524    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-service-ca\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.207573 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.207548    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-oauth-serving-cert\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.207573 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.207568    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e923894-d919-4eaf-a313-0643ba059228-console-serving-cert\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.207927 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.207584    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-trusted-ca-bundle\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.207927 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.207716    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcf84\" (UniqueName: \"kubernetes.io/projected/8e923894-d919-4eaf-a313-0643ba059228-kube-api-access-kcf84\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.207927 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.207799    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e923894-d919-4eaf-a313-0643ba059228-console-oauth-config\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.208381 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.208353    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-console-config\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.208496 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.208455    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-oauth-serving-cert\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.208620 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.208489    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-service-ca\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.208663 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.208583    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e923894-d919-4eaf-a313-0643ba059228-trusted-ca-bundle\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.210445 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.210418    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e923894-d919-4eaf-a313-0643ba059228-console-serving-cert\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.210445 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.210440    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e923894-d919-4eaf-a313-0643ba059228-console-oauth-config\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.216981 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.216961    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcf84\" (UniqueName: \"kubernetes.io/projected/8e923894-d919-4eaf-a313-0643ba059228-kube-api-access-kcf84\") pod \"console-fc976d4b7-m2j4h\" (UID: \"8e923894-d919-4eaf-a313-0643ba059228\") " pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.314367 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.314329    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:43.587798 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.587776    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc976d4b7-m2j4h"]
Sep 17 18:45:43.590270 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:45:43.590246    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e923894_d919_4eaf_a313_0643ba059228.slice/crio-605d9fe93b2d2636290df5b04675b6167b2a3f661fba84e4cc04af70afc7b812 WatchSource:0}: Error finding container 605d9fe93b2d2636290df5b04675b6167b2a3f661fba84e4cc04af70afc7b812: Status 404 returned error can't find the container with id 605d9fe93b2d2636290df5b04675b6167b2a3f661fba84e4cc04af70afc7b812
Sep 17 18:45:43.934679 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.934581    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc976d4b7-m2j4h" event={"ID":"8e923894-d919-4eaf-a313-0643ba059228","Type":"ContainerStarted","Data":"27de541540f192be4e1baae19afca2007af406b6fa6255cff012c14a721acc1a"}
Sep 17 18:45:43.934679 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.934648    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc976d4b7-m2j4h" event={"ID":"8e923894-d919-4eaf-a313-0643ba059228","Type":"ContainerStarted","Data":"605d9fe93b2d2636290df5b04675b6167b2a3f661fba84e4cc04af70afc7b812"}
Sep 17 18:45:43.954451 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:43.954400    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fc976d4b7-m2j4h" podStartSLOduration=1.95438211 podStartE2EDuration="1.95438211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:43.95256739 +0000 UTC m=+364.812392075" watchObservedRunningTime="2026-09-17 18:45:43.95438211 +0000 UTC m=+364.814206757"
Sep 17 18:45:45.948275 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:45.948233    2586 generic.go:356] "Generic (PLEG): container finished" podID="d907546c-6c1a-4090-91d5-9b8f23dedd4a" containerID="8f873141998e4b0acb61ca7984b8d7f1648cb4d89d1abede308a97c8af58d2f6" exitCode=0
Sep 17 18:45:45.948650 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:45.948295    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6" event={"ID":"d907546c-6c1a-4090-91d5-9b8f23dedd4a","Type":"ContainerDied","Data":"8f873141998e4b0acb61ca7984b8d7f1648cb4d89d1abede308a97c8af58d2f6"}
Sep 17 18:45:46.955750 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:46.955711    2586 generic.go:356] "Generic (PLEG): container finished" podID="d907546c-6c1a-4090-91d5-9b8f23dedd4a" containerID="b4c42acef19e92daaf73ac3791e3a4509c436bf808cbebc4e9f3cfb42d06fe6a" exitCode=0
Sep 17 18:45:46.956158 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:46.955795    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6" event={"ID":"d907546c-6c1a-4090-91d5-9b8f23dedd4a","Type":"ContainerDied","Data":"b4c42acef19e92daaf73ac3791e3a4509c436bf808cbebc4e9f3cfb42d06fe6a"}
Sep 17 18:45:48.086037 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.086007    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:48.152038 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.151995    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-util\") pod \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") "
Sep 17 18:45:48.152225 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.152106    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-bundle\") pod \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") "
Sep 17 18:45:48.152225 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.152125    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlpjq\" (UniqueName: \"kubernetes.io/projected/d907546c-6c1a-4090-91d5-9b8f23dedd4a-kube-api-access-xlpjq\") pod \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\" (UID: \"d907546c-6c1a-4090-91d5-9b8f23dedd4a\") "
Sep 17 18:45:48.152764 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.152735    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-bundle" (OuterVolumeSpecName: "bundle") pod "d907546c-6c1a-4090-91d5-9b8f23dedd4a" (UID: "d907546c-6c1a-4090-91d5-9b8f23dedd4a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:48.154352 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.154329    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d907546c-6c1a-4090-91d5-9b8f23dedd4a-kube-api-access-xlpjq" (OuterVolumeSpecName: "kube-api-access-xlpjq") pod "d907546c-6c1a-4090-91d5-9b8f23dedd4a" (UID: "d907546c-6c1a-4090-91d5-9b8f23dedd4a"). InnerVolumeSpecName "kube-api-access-xlpjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:48.156402 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.156376    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-util" (OuterVolumeSpecName: "util") pod "d907546c-6c1a-4090-91d5-9b8f23dedd4a" (UID: "d907546c-6c1a-4090-91d5-9b8f23dedd4a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:48.252946 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.252867    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:45:48.252946 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.252893    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlpjq\" (UniqueName: \"kubernetes.io/projected/d907546c-6c1a-4090-91d5-9b8f23dedd4a-kube-api-access-xlpjq\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:45:48.252946 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.252904    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d907546c-6c1a-4090-91d5-9b8f23dedd4a-util\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:45:48.965807 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.965765    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6" event={"ID":"d907546c-6c1a-4090-91d5-9b8f23dedd4a","Type":"ContainerDied","Data":"88a52d4529f3cdce50bcaac6ff94895c052a7b01c2d21fc11d180b0b09e80b19"}
Sep 17 18:45:48.965807 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.965813    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88a52d4529f3cdce50bcaac6ff94895c052a7b01c2d21fc11d180b0b09e80b19"
Sep 17 18:45:48.965807 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:48.965813    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kvhn6"
Sep 17 18:45:53.315496 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:53.315432    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:53.316008 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:53.315567    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:53.320327 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:53.320302    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:53.991311 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:53.991281    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fc976d4b7-m2j4h"
Sep 17 18:45:54.069814 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:54.069783    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d899b4755-c2kjf"]
Sep 17 18:45:55.027762 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.027730    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"]
Sep 17 18:45:55.028132 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.028059    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d907546c-6c1a-4090-91d5-9b8f23dedd4a" containerName="util"
Sep 17 18:45:55.028132 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.028071    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d907546c-6c1a-4090-91d5-9b8f23dedd4a" containerName="util"
Sep 17 18:45:55.028132 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.028094    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d907546c-6c1a-4090-91d5-9b8f23dedd4a" containerName="pull"
Sep 17 18:45:55.028132 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.028099    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d907546c-6c1a-4090-91d5-9b8f23dedd4a" containerName="pull"
Sep 17 18:45:55.028132 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.028111    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d907546c-6c1a-4090-91d5-9b8f23dedd4a" containerName="extract"
Sep 17 18:45:55.028132 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.028119    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d907546c-6c1a-4090-91d5-9b8f23dedd4a" containerName="extract"
Sep 17 18:45:55.028327 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.028177    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d907546c-6c1a-4090-91d5-9b8f23dedd4a" containerName="extract"
Sep 17 18:45:55.030156 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.030141    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"
Sep 17 18:45:55.035699 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.035678    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 18:45:55.035862 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.035718    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:45:55.035862 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.035720    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8dhzs\""
Sep 17 18:45:55.052784 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.052755    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"]
Sep 17 18:45:55.110887 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.110851    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/051ae952-bbf4-4626-bc1a-c175bcb27bc3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-fr6c7\" (UID: \"051ae952-bbf4-4626-bc1a-c175bcb27bc3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"
Sep 17 18:45:55.111079 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.110907    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5jp\" (UniqueName: \"kubernetes.io/projected/051ae952-bbf4-4626-bc1a-c175bcb27bc3-kube-api-access-7b5jp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-fr6c7\" (UID: \"051ae952-bbf4-4626-bc1a-c175bcb27bc3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"
Sep 17 18:45:55.211860 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.211801    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/051ae952-bbf4-4626-bc1a-c175bcb27bc3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-fr6c7\" (UID: \"051ae952-bbf4-4626-bc1a-c175bcb27bc3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"
Sep 17 18:45:55.212024 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.211869    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5jp\" (UniqueName: \"kubernetes.io/projected/051ae952-bbf4-4626-bc1a-c175bcb27bc3-kube-api-access-7b5jp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-fr6c7\" (UID: \"051ae952-bbf4-4626-bc1a-c175bcb27bc3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"
Sep 17 18:45:55.212183 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.212165    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/051ae952-bbf4-4626-bc1a-c175bcb27bc3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-fr6c7\" (UID: \"051ae952-bbf4-4626-bc1a-c175bcb27bc3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"
Sep 17 18:45:55.224243 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.224209    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5jp\" (UniqueName: \"kubernetes.io/projected/051ae952-bbf4-4626-bc1a-c175bcb27bc3-kube-api-access-7b5jp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-fr6c7\" (UID: \"051ae952-bbf4-4626-bc1a-c175bcb27bc3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"
Sep 17 18:45:55.339472 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.339439    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"
Sep 17 18:45:55.618292 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.618253    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7"]
Sep 17 18:45:55.620594 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:45:55.620561    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051ae952_bbf4_4626_bc1a_c175bcb27bc3.slice/crio-9743617e1ed24aa4080521ba1e510527d0a9f2ff0e9d1afad112ee154945da76 WatchSource:0}: Error finding container 9743617e1ed24aa4080521ba1e510527d0a9f2ff0e9d1afad112ee154945da76: Status 404 returned error can't find the container with id 9743617e1ed24aa4080521ba1e510527d0a9f2ff0e9d1afad112ee154945da76
Sep 17 18:45:55.998011 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:55.997919    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7" event={"ID":"051ae952-bbf4-4626-bc1a-c175bcb27bc3","Type":"ContainerStarted","Data":"9743617e1ed24aa4080521ba1e510527d0a9f2ff0e9d1afad112ee154945da76"}
Sep 17 18:45:58.011575 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.011527    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7" event={"ID":"051ae952-bbf4-4626-bc1a-c175bcb27bc3","Type":"ContainerStarted","Data":"b1d74e4f6ff4524d8fff35b073f34fc7a2a89c30baf7d294e341633b53d53053"}
Sep 17 18:45:58.036068 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.036013    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-fr6c7" podStartSLOduration=1.3262331299999999 podStartE2EDuration="3.035996281s" totalImagesPullingTime="1.709763151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:58.034865849 +0000 UTC m=+378.894690578" watchObservedRunningTime="2026-09-17 18:45:58.035996281 +0000 UTC m=+378.895820929"
Sep 17 18:45:58.408251 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.408219    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gdfqp"]
Sep 17 18:45:58.410799 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.410783    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp"
Sep 17 18:45:58.413896 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.413875    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-52n84\""
Sep 17 18:45:58.414400 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.414382    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 18:45:58.414482 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.414419    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 18:45:58.427260 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.427238    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gdfqp"]
Sep 17 18:45:58.541062 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.541021    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4475d46c-da46-437b-acb8-a77353d718e6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gdfqp\" (UID: \"4475d46c-da46-437b-acb8-a77353d718e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp"
Sep 17 18:45:58.541062 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.541063    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2l2\" (UniqueName: \"kubernetes.io/projected/4475d46c-da46-437b-acb8-a77353d718e6-kube-api-access-pq2l2\") pod \"cert-manager-webhook-6486ffd997-gdfqp\" (UID: \"4475d46c-da46-437b-acb8-a77353d718e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp"
Sep 17 18:45:58.642441 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.642399    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4475d46c-da46-437b-acb8-a77353d718e6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gdfqp\" (UID: \"4475d46c-da46-437b-acb8-a77353d718e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp"
Sep 17 18:45:58.642441 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.642436    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2l2\" (UniqueName: \"kubernetes.io/projected/4475d46c-da46-437b-acb8-a77353d718e6-kube-api-access-pq2l2\") pod \"cert-manager-webhook-6486ffd997-gdfqp\" (UID: \"4475d46c-da46-437b-acb8-a77353d718e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp"
Sep 17 18:45:58.651718 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.651687    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4475d46c-da46-437b-acb8-a77353d718e6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gdfqp\" (UID: \"4475d46c-da46-437b-acb8-a77353d718e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp"
Sep 17 18:45:58.651860 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.651803    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2l2\" (UniqueName: \"kubernetes.io/projected/4475d46c-da46-437b-acb8-a77353d718e6-kube-api-access-pq2l2\") pod \"cert-manager-webhook-6486ffd997-gdfqp\" (UID: \"4475d46c-da46-437b-acb8-a77353d718e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp"
Sep 17 18:45:58.726400 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:58.726316    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp"
Sep 17 18:45:59.005897 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:59.005807    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gdfqp"]
Sep 17 18:45:59.009013 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:45:59.008985    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4475d46c_da46_437b_acb8_a77353d718e6.slice/crio-b23103551d24d3d07502e23a8aedf106c7baafe57c60bdf112c2ecec19802be8 WatchSource:0}: Error finding container b23103551d24d3d07502e23a8aedf106c7baafe57c60bdf112c2ecec19802be8: Status 404 returned error can't find the container with id b23103551d24d3d07502e23a8aedf106c7baafe57c60bdf112c2ecec19802be8
Sep 17 18:45:59.017443 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:45:59.017416    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp" event={"ID":"4475d46c-da46-437b-acb8-a77353d718e6","Type":"ContainerStarted","Data":"b23103551d24d3d07502e23a8aedf106c7baafe57c60bdf112c2ecec19802be8"}
Sep 17 18:46:02.033188 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:02.033102    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp" event={"ID":"4475d46c-da46-437b-acb8-a77353d718e6","Type":"ContainerStarted","Data":"1e91deae88ec8618add4e8e819c315468d83de2b507ade37e43914f9d98a9417"}
Sep 17 18:46:02.033570 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:02.033218    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp"
Sep 17 18:46:02.055100 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:02.055048    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp" podStartSLOduration=1.312725905 podStartE2EDuration="4.055032935s" totalImagesPullingTime="2.74230703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:02.053022352 +0000 UTC m=+382.912847010" watchObservedRunningTime="2026-09-17 18:46:02.055032935 +0000 UTC m=+382.914857582"
Sep 17 18:46:04.102098 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.102063    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"]
Sep 17 18:46:04.104519 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.104500    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.108076 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.108049    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:46:04.108178 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.108049    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xl97j\""
Sep 17 18:46:04.108927 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.108907    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:46:04.117640 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.117615    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"]
Sep 17 18:46:04.191921 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.191882    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.191921 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.191925    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlfk2\" (UniqueName: \"kubernetes.io/projected/906e5dcf-966d-43b5-a60d-089db526c902-kube-api-access-nlfk2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.192131 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.191955    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.292932 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.292895    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.293099 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.292942    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlfk2\" (UniqueName: \"kubernetes.io/projected/906e5dcf-966d-43b5-a60d-089db526c902-kube-api-access-nlfk2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.293099 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.293060    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.293334 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.293314    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.293402 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.293355    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.302506 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.302475    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlfk2\" (UniqueName: \"kubernetes.io/projected/906e5dcf-966d-43b5-a60d-089db526c902-kube-api-access-nlfk2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.413680 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.413584    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:04.687223 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:04.687144    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"]
Sep 17 18:46:04.690724 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:46:04.690695    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906e5dcf_966d_43b5_a60d_089db526c902.slice/crio-63d8ae33e145aedf1982f95f2faac03e9d9f1af922458e2a74c76f57e0249f0e WatchSource:0}: Error finding container 63d8ae33e145aedf1982f95f2faac03e9d9f1af922458e2a74c76f57e0249f0e: Status 404 returned error can't find the container with id 63d8ae33e145aedf1982f95f2faac03e9d9f1af922458e2a74c76f57e0249f0e
Sep 17 18:46:05.048494 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:05.048408    2586 generic.go:356] "Generic (PLEG): container finished" podID="906e5dcf-966d-43b5-a60d-089db526c902" containerID="1c3b8d2d5b405f26614262943fe6d8d4280929087b52e0e15c21e7cc89f26790" exitCode=0
Sep 17 18:46:05.048650 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:05.048492    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn" event={"ID":"906e5dcf-966d-43b5-a60d-089db526c902","Type":"ContainerDied","Data":"1c3b8d2d5b405f26614262943fe6d8d4280929087b52e0e15c21e7cc89f26790"}
Sep 17 18:46:05.048650 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:05.048543    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn" event={"ID":"906e5dcf-966d-43b5-a60d-089db526c902","Type":"ContainerStarted","Data":"63d8ae33e145aedf1982f95f2faac03e9d9f1af922458e2a74c76f57e0249f0e"}
Sep 17 18:46:07.059667 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:07.059637    2586 generic.go:356] "Generic (PLEG): container finished" podID="906e5dcf-966d-43b5-a60d-089db526c902" containerID="cb420edc7c4eaa2f66bfcec2d58c075a408a0a6d6ebe4a9175a4f8bf4813e403" exitCode=0
Sep 17 18:46:07.060079 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:07.059723    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn" event={"ID":"906e5dcf-966d-43b5-a60d-089db526c902","Type":"ContainerDied","Data":"cb420edc7c4eaa2f66bfcec2d58c075a408a0a6d6ebe4a9175a4f8bf4813e403"}
Sep 17 18:46:08.040097 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:08.040063    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-gdfqp"
Sep 17 18:46:08.066112 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:08.066080    2586 generic.go:356] "Generic (PLEG): container finished" podID="906e5dcf-966d-43b5-a60d-089db526c902" containerID="80a96d1e378e10474a9737018c47146ae85ffa23f01f93c82e434589df407950" exitCode=0
Sep 17 18:46:08.066468 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:08.066135    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn" event={"ID":"906e5dcf-966d-43b5-a60d-089db526c902","Type":"ContainerDied","Data":"80a96d1e378e10474a9737018c47146ae85ffa23f01f93c82e434589df407950"}
Sep 17 18:46:09.199062 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:09.199037    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:09.338953 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:09.338865    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-bundle\") pod \"906e5dcf-966d-43b5-a60d-089db526c902\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") "
Sep 17 18:46:09.339118 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:09.338961    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-util\") pod \"906e5dcf-966d-43b5-a60d-089db526c902\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") "
Sep 17 18:46:09.339118 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:09.338986    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlfk2\" (UniqueName: \"kubernetes.io/projected/906e5dcf-966d-43b5-a60d-089db526c902-kube-api-access-nlfk2\") pod \"906e5dcf-966d-43b5-a60d-089db526c902\" (UID: \"906e5dcf-966d-43b5-a60d-089db526c902\") "
Sep 17 18:46:09.339311 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:09.339285    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-bundle" (OuterVolumeSpecName: "bundle") pod "906e5dcf-966d-43b5-a60d-089db526c902" (UID: "906e5dcf-966d-43b5-a60d-089db526c902"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:09.341285 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:09.341258    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906e5dcf-966d-43b5-a60d-089db526c902-kube-api-access-nlfk2" (OuterVolumeSpecName: "kube-api-access-nlfk2") pod "906e5dcf-966d-43b5-a60d-089db526c902" (UID: "906e5dcf-966d-43b5-a60d-089db526c902"). InnerVolumeSpecName "kube-api-access-nlfk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:09.344176 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:09.344146    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-util" (OuterVolumeSpecName: "util") pod "906e5dcf-966d-43b5-a60d-089db526c902" (UID: "906e5dcf-966d-43b5-a60d-089db526c902"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:09.440131 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:09.440080    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-util\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:09.440131 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:09.440127    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlfk2\" (UniqueName: \"kubernetes.io/projected/906e5dcf-966d-43b5-a60d-089db526c902-kube-api-access-nlfk2\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:09.440131 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:09.440141    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/906e5dcf-966d-43b5-a60d-089db526c902-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:10.077362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:10.077324    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn" event={"ID":"906e5dcf-966d-43b5-a60d-089db526c902","Type":"ContainerDied","Data":"63d8ae33e145aedf1982f95f2faac03e9d9f1af922458e2a74c76f57e0249f0e"}
Sep 17 18:46:10.077362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:10.077357    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftlmtn"
Sep 17 18:46:10.077362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:10.077371    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63d8ae33e145aedf1982f95f2faac03e9d9f1af922458e2a74c76f57e0249f0e"
Sep 17 18:46:14.421814 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.421778    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"]
Sep 17 18:46:14.422301 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.422096    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906e5dcf-966d-43b5-a60d-089db526c902" containerName="extract"
Sep 17 18:46:14.422301 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.422107    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="906e5dcf-966d-43b5-a60d-089db526c902" containerName="extract"
Sep 17 18:46:14.422301 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.422123    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906e5dcf-966d-43b5-a60d-089db526c902" containerName="util"
Sep 17 18:46:14.422301 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.422128    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="906e5dcf-966d-43b5-a60d-089db526c902" containerName="util"
Sep 17 18:46:14.422301 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.422134    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906e5dcf-966d-43b5-a60d-089db526c902" containerName="pull"
Sep 17 18:46:14.422301 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.422140    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="906e5dcf-966d-43b5-a60d-089db526c902" containerName="pull"
Sep 17 18:46:14.422301 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.422194    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="906e5dcf-966d-43b5-a60d-089db526c902" containerName="extract"
Sep 17 18:46:14.424344 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.424328    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"
Sep 17 18:46:14.428884 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.428863    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:46:14.429937 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.429921    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-q6rrx\""
Sep 17 18:46:14.430000 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.429933    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 18:46:14.440297 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.440269    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"]
Sep 17 18:46:14.579080 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.579038    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de36c1d9-f8cb-4780-8a46-410602b7c54c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-77dw8\" (UID: \"de36c1d9-f8cb-4780-8a46-410602b7c54c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"
Sep 17 18:46:14.579300 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.579130    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8fbg\" (UniqueName: \"kubernetes.io/projected/de36c1d9-f8cb-4780-8a46-410602b7c54c-kube-api-access-b8fbg\") pod \"openshift-lws-operator-fd8ccff4c-77dw8\" (UID: \"de36c1d9-f8cb-4780-8a46-410602b7c54c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"
Sep 17 18:46:14.680096 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.680007    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de36c1d9-f8cb-4780-8a46-410602b7c54c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-77dw8\" (UID: \"de36c1d9-f8cb-4780-8a46-410602b7c54c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"
Sep 17 18:46:14.680096 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.680061    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8fbg\" (UniqueName: \"kubernetes.io/projected/de36c1d9-f8cb-4780-8a46-410602b7c54c-kube-api-access-b8fbg\") pod \"openshift-lws-operator-fd8ccff4c-77dw8\" (UID: \"de36c1d9-f8cb-4780-8a46-410602b7c54c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"
Sep 17 18:46:14.680390 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.680368    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de36c1d9-f8cb-4780-8a46-410602b7c54c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-77dw8\" (UID: \"de36c1d9-f8cb-4780-8a46-410602b7c54c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"
Sep 17 18:46:14.693347 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.693317    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8fbg\" (UniqueName: \"kubernetes.io/projected/de36c1d9-f8cb-4780-8a46-410602b7c54c-kube-api-access-b8fbg\") pod \"openshift-lws-operator-fd8ccff4c-77dw8\" (UID: \"de36c1d9-f8cb-4780-8a46-410602b7c54c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"
Sep 17 18:46:14.733565 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:14.733527    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"
Sep 17 18:46:15.003092 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:15.003068    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8"]
Sep 17 18:46:15.005517 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:46:15.005475    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde36c1d9_f8cb_4780_8a46_410602b7c54c.slice/crio-2eb8dd593ee5cf678d0f385aa5743a446475d0188c76a3eb4c648a9fc968937c WatchSource:0}: Error finding container 2eb8dd593ee5cf678d0f385aa5743a446475d0188c76a3eb4c648a9fc968937c: Status 404 returned error can't find the container with id 2eb8dd593ee5cf678d0f385aa5743a446475d0188c76a3eb4c648a9fc968937c
Sep 17 18:46:15.102103 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:15.102066    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8" event={"ID":"de36c1d9-f8cb-4780-8a46-410602b7c54c","Type":"ContainerStarted","Data":"2eb8dd593ee5cf678d0f385aa5743a446475d0188c76a3eb4c648a9fc968937c"}
Sep 17 18:46:17.115336 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:17.115301    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8" event={"ID":"de36c1d9-f8cb-4780-8a46-410602b7c54c","Type":"ContainerStarted","Data":"93fca8aed485439f70e5dce33b9e3d26bbe667784d850e73e56c39a7a34c3ba6"}
Sep 17 18:46:17.153462 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:17.153409    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-77dw8" podStartSLOduration=1.449140522 podStartE2EDuration="3.153391713s" totalImagesPullingTime="1.704251191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:17.152272225 +0000 UTC m=+398.012096874" watchObservedRunningTime="2026-09-17 18:46:17.153391713 +0000 UTC m=+398.013216361"
Sep 17 18:46:19.089878 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.089816    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d899b4755-c2kjf" podUID="4d8d5258-b30f-4796-9e50-da80e93f132b" containerName="console" containerID="cri-o://0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e" gracePeriod=15
Sep 17 18:46:19.352682 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.352611    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:46:19.525649 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.525617    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-serving-cert\") pod \"4d8d5258-b30f-4796-9e50-da80e93f132b\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") "
Sep 17 18:46:19.525876 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.525655    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-service-ca\") pod \"4d8d5258-b30f-4796-9e50-da80e93f132b\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") "
Sep 17 18:46:19.525876 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.525679    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-oauth-serving-cert\") pod \"4d8d5258-b30f-4796-9e50-da80e93f132b\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") "
Sep 17 18:46:19.525876 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.525701    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzj9l\" (UniqueName: \"kubernetes.io/projected/4d8d5258-b30f-4796-9e50-da80e93f132b-kube-api-access-gzj9l\") pod \"4d8d5258-b30f-4796-9e50-da80e93f132b\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") "
Sep 17 18:46:19.525876 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.525769    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-trusted-ca-bundle\") pod \"4d8d5258-b30f-4796-9e50-da80e93f132b\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") "
Sep 17 18:46:19.525876 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.525802    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-oauth-config\") pod \"4d8d5258-b30f-4796-9e50-da80e93f132b\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") "
Sep 17 18:46:19.525876 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.525852    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-console-config\") pod \"4d8d5258-b30f-4796-9e50-da80e93f132b\" (UID: \"4d8d5258-b30f-4796-9e50-da80e93f132b\") "
Sep 17 18:46:19.526154 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.526134    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4d8d5258-b30f-4796-9e50-da80e93f132b" (UID: "4d8d5258-b30f-4796-9e50-da80e93f132b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:19.526210 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.526165    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-service-ca" (OuterVolumeSpecName: "service-ca") pod "4d8d5258-b30f-4796-9e50-da80e93f132b" (UID: "4d8d5258-b30f-4796-9e50-da80e93f132b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:19.526368 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.526337    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-console-config" (OuterVolumeSpecName: "console-config") pod "4d8d5258-b30f-4796-9e50-da80e93f132b" (UID: "4d8d5258-b30f-4796-9e50-da80e93f132b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:19.526432 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.526364    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4d8d5258-b30f-4796-9e50-da80e93f132b" (UID: "4d8d5258-b30f-4796-9e50-da80e93f132b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:19.528232 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.528200    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4d8d5258-b30f-4796-9e50-da80e93f132b" (UID: "4d8d5258-b30f-4796-9e50-da80e93f132b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:19.528509 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.528486    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8d5258-b30f-4796-9e50-da80e93f132b-kube-api-access-gzj9l" (OuterVolumeSpecName: "kube-api-access-gzj9l") pod "4d8d5258-b30f-4796-9e50-da80e93f132b" (UID: "4d8d5258-b30f-4796-9e50-da80e93f132b"). InnerVolumeSpecName "kube-api-access-gzj9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:19.528509 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.528492    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4d8d5258-b30f-4796-9e50-da80e93f132b" (UID: "4d8d5258-b30f-4796-9e50-da80e93f132b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:19.626958 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.626865    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-trusted-ca-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:19.626958 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.626903    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-oauth-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:19.626958 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.626913    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-console-config\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:19.626958 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.626921    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d8d5258-b30f-4796-9e50-da80e93f132b-console-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:19.626958 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.626929    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-service-ca\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:19.626958 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.626937    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d8d5258-b30f-4796-9e50-da80e93f132b-oauth-serving-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:19.626958 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:19.626945    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzj9l\" (UniqueName: \"kubernetes.io/projected/4d8d5258-b30f-4796-9e50-da80e93f132b-kube-api-access-gzj9l\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:20.130527 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:20.130487    2586 generic.go:356] "Generic (PLEG): container finished" podID="4d8d5258-b30f-4796-9e50-da80e93f132b" containerID="0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e" exitCode=0
Sep 17 18:46:20.130983 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:20.130542    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d899b4755-c2kjf"
Sep 17 18:46:20.130983 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:20.130573    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d899b4755-c2kjf" event={"ID":"4d8d5258-b30f-4796-9e50-da80e93f132b","Type":"ContainerDied","Data":"0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e"}
Sep 17 18:46:20.130983 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:20.130624    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d899b4755-c2kjf" event={"ID":"4d8d5258-b30f-4796-9e50-da80e93f132b","Type":"ContainerDied","Data":"43c18e3e0172be7f2ec935a7c6f4b11df5a61e762919cc345d6b19f114b8ac86"}
Sep 17 18:46:20.130983 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:20.130644    2586 scope.go:117] "RemoveContainer" containerID="0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e"
Sep 17 18:46:20.139953 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:20.139934    2586 scope.go:117] "RemoveContainer" containerID="0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e"
Sep 17 18:46:20.140276 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:46:20.140256    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e\": container with ID starting with 0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e not found: ID does not exist" containerID="0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e"
Sep 17 18:46:20.140338 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:20.140290    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e"} err="failed to get container status \"0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e\": rpc error: code = NotFound desc = could not find container \"0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e\": container with ID starting with 0603693d35478a53e298850b149720b92b5381340621ee39454dad90c629c35e not found: ID does not exist"
Sep 17 18:46:20.152582 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:20.152548    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d899b4755-c2kjf"]
Sep 17 18:46:20.156173 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:20.156145    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d899b4755-c2kjf"]
Sep 17 18:46:21.458077 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:21.458044    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8d5258-b30f-4796-9e50-da80e93f132b" path="/var/lib/kubelet/pods/4d8d5258-b30f-4796-9e50-da80e93f132b/volumes"
Sep 17 18:46:27.026754 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.026709    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"]
Sep 17 18:46:27.027226 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.027165    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8d5258-b30f-4796-9e50-da80e93f132b" containerName="console"
Sep 17 18:46:27.027226 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.027182    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8d5258-b30f-4796-9e50-da80e93f132b" containerName="console"
Sep 17 18:46:27.027351 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.027290    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8d5258-b30f-4796-9e50-da80e93f132b" containerName="console"
Sep 17 18:46:27.030327 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.030304    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.033617 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.033590    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 18:46:27.033991 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.033973    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 18:46:27.034055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.034041    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lgfmw\""
Sep 17 18:46:27.034355 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.034340    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:46:27.034689 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.034672    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 18:46:27.034917 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.034892    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 18:46:27.035152 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.035140    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 18:46:27.049233 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.049203    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"]
Sep 17 18:46:27.076111 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.076083    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.076111 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.076113    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.076318 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.076132    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.076318 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.076149    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.076318 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.076233    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.076318 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.076283    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw26j\" (UniqueName: \"kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-kube-api-access-jw26j\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.076318 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.076300    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.176676 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.176644    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.176911 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.176686    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw26j\" (UniqueName: \"kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-kube-api-access-jw26j\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.176911 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.176717    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.176911 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.176772    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.176911 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.176797    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.176911 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.176854    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.176911 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.176880    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.177583 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.177560    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.179478 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.179457    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.179582 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.179556    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.179707 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.179690    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.179811 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.179793    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.185650 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.185628    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.185917 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.185898    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw26j\" (UniqueName: \"kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-kube-api-access-jw26j\") pod \"istiod-openshift-gateway-7b7995d5cd-zw26s\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.339888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.339859    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:27.628223 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:27.628156    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"]
Sep 17 18:46:27.631099 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:46:27.631048    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9baf987_4ae7_42dc_8c0a_9249b2dee96f.slice/crio-a2245d0ae464b6dc4cd33a89f266cf36798fc895b8b5ac4680e4a39f97242cc4 WatchSource:0}: Error finding container a2245d0ae464b6dc4cd33a89f266cf36798fc895b8b5ac4680e4a39f97242cc4: Status 404 returned error can't find the container with id a2245d0ae464b6dc4cd33a89f266cf36798fc895b8b5ac4680e4a39f97242cc4
Sep 17 18:46:28.168086 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:28.168051    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s" event={"ID":"e9baf987-4ae7-42dc-8c0a-9249b2dee96f","Type":"ContainerStarted","Data":"a2245d0ae464b6dc4cd33a89f266cf36798fc895b8b5ac4680e4a39f97242cc4"}
Sep 17 18:46:30.326458 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:30.326419    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:46:30.326708 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:30.326497    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:46:31.186996 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:31.186957    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s" event={"ID":"e9baf987-4ae7-42dc-8c0a-9249b2dee96f","Type":"ContainerStarted","Data":"dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59"}
Sep 17 18:46:31.187290 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:31.187269    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:31.189164 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:31.189131    2586 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-zw26s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 18:46:31.189270 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:31.189188    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s" podUID="e9baf987-4ae7-42dc-8c0a-9249b2dee96f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:46:31.212311 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:31.212262    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s" podStartSLOduration=1.519691761 podStartE2EDuration="4.212246199s" totalImagesPullingTime="2.692554438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:31.211149551 +0000 UTC m=+412.070974220" watchObservedRunningTime="2026-09-17 18:46:31.212246199 +0000 UTC m=+412.072070847"
Sep 17 18:46:32.192382 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:32.192353    2586 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-zw26s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 18:46:32.192885 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:32.192407    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s" podUID="e9baf987-4ae7-42dc-8c0a-9249b2dee96f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:46:35.193023 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:35.192990    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:46:45.919932 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:45.919894    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"]
Sep 17 18:46:45.925171 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:45.925147    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:45.930358 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:45.930330    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xl97j\""
Sep 17 18:46:45.930358 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:45.930356    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:46:45.930705 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:45.930689    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:46:45.946039 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:45.946012    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"]
Sep 17 18:46:46.016786 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.016751    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"]
Sep 17 18:46:46.019802 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.019769    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.029605 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.029579    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"]
Sep 17 18:46:46.036403 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.036377    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5ls\" (UniqueName: \"kubernetes.io/projected/99443a86-943e-4b69-9867-df6117bb33fa-kube-api-access-mx5ls\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:46.036530 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.036442    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:46.036530 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.036478    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:46.106609 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.106569    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"]
Sep 17 18:46:46.109196 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.109173    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.125251 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.125225    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"]
Sep 17 18:46:46.137700 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.137671    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dxbr\" (UniqueName: \"kubernetes.io/projected/09f4dac5-8796-44a4-877d-ecc640c815dc-kube-api-access-2dxbr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.137880 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.137714    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5ls\" (UniqueName: \"kubernetes.io/projected/99443a86-943e-4b69-9867-df6117bb33fa-kube-api-access-mx5ls\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:46.137880 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.137789    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.137880 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.137824    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:46.138015 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.137879    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.138015 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.137926    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:46.138172 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.138153    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:46.138213 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.138190    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:46.157522 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.157487    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5ls\" (UniqueName: \"kubernetes.io/projected/99443a86-943e-4b69-9867-df6117bb33fa-kube-api-access-mx5ls\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:46.234344 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.234234    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:46.236992 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.236971    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"]
Sep 17 18:46:46.238942 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.238922    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.239034 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.238952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dxbr\" (UniqueName: \"kubernetes.io/projected/09f4dac5-8796-44a4-877d-ecc640c815dc-kube-api-access-2dxbr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.239034 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.238989    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lkk\" (UniqueName: \"kubernetes.io/projected/a33b9774-3193-418f-b8d9-e65403df617a-kube-api-access-b5lkk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.239138 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.239075    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.239138 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.239128    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.239241 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.239165    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.242861 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.239876    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.246499 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.246464    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.247035 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.247013    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.255804 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.255769    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"]
Sep 17 18:46:46.262401 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.262334    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dxbr\" (UniqueName: \"kubernetes.io/projected/09f4dac5-8796-44a4-877d-ecc640c815dc-kube-api-access-2dxbr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.329955 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.329920    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:46.339706 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.339675    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.339882 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.339726    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.339882 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.339745    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.339882 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.339764    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftl2r\" (UniqueName: \"kubernetes.io/projected/df2701f0-3c8b-43ef-ac58-11550fb6618e-kube-api-access-ftl2r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.339882 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.339797    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lkk\" (UniqueName: \"kubernetes.io/projected/a33b9774-3193-418f-b8d9-e65403df617a-kube-api-access-b5lkk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.339882 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.339828    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.340152 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.340134    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.340196 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.340153    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.366014 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.365973    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lkk\" (UniqueName: \"kubernetes.io/projected/a33b9774-3193-418f-b8d9-e65403df617a-kube-api-access-b5lkk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.418417 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.418376    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:46.441315 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.441274    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.441315 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.441313    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.441512 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.441339    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftl2r\" (UniqueName: \"kubernetes.io/projected/df2701f0-3c8b-43ef-ac58-11550fb6618e-kube-api-access-ftl2r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.441658 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.441637    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.441860 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.441821    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.466115 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.466058    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftl2r\" (UniqueName: \"kubernetes.io/projected/df2701f0-3c8b-43ef-ac58-11550fb6618e-kube-api-access-ftl2r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.542296 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.542185    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"]
Sep 17 18:46:46.546017 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:46:46.545983    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99443a86_943e_4b69_9867_df6117bb33fa.slice/crio-ccb5b4dda838abbde36efbcccf5a115140d63a0528e352d2ccdca12c344c21e7 WatchSource:0}: Error finding container ccb5b4dda838abbde36efbcccf5a115140d63a0528e352d2ccdca12c344c21e7: Status 404 returned error can't find the container with id ccb5b4dda838abbde36efbcccf5a115140d63a0528e352d2ccdca12c344c21e7
Sep 17 18:46:46.581502 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.581476    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:46:46.616635 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.616598    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"]
Sep 17 18:46:46.620087 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:46:46.620051    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f4dac5_8796_44a4_877d_ecc640c815dc.slice/crio-1e9ccd47a1763f2688cf9d4c3655f31c31aa1a72cea7d0bfb13f59a032fa5a20 WatchSource:0}: Error finding container 1e9ccd47a1763f2688cf9d4c3655f31c31aa1a72cea7d0bfb13f59a032fa5a20: Status 404 returned error can't find the container with id 1e9ccd47a1763f2688cf9d4c3655f31c31aa1a72cea7d0bfb13f59a032fa5a20
Sep 17 18:46:46.699652 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.699608    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"]
Sep 17 18:46:46.714860 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:46:46.714814    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33b9774_3193_418f_b8d9_e65403df617a.slice/crio-cdb02a69fa1fe41c4d774e1534ad1ae0f812c2a4d4bc2f12326f2b0ef4f99418 WatchSource:0}: Error finding container cdb02a69fa1fe41c4d774e1534ad1ae0f812c2a4d4bc2f12326f2b0ef4f99418: Status 404 returned error can't find the container with id cdb02a69fa1fe41c4d774e1534ad1ae0f812c2a4d4bc2f12326f2b0ef4f99418
Sep 17 18:46:46.867314 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:46.867291    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"]
Sep 17 18:46:46.892825 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:46:46.892793    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2701f0_3c8b_43ef_ac58_11550fb6618e.slice/crio-71f562b132537f087f07c3824cd6bb5f70d94db82300a06555cddcfadcf4b970 WatchSource:0}: Error finding container 71f562b132537f087f07c3824cd6bb5f70d94db82300a06555cddcfadcf4b970: Status 404 returned error can't find the container with id 71f562b132537f087f07c3824cd6bb5f70d94db82300a06555cddcfadcf4b970
Sep 17 18:46:47.269384 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.269286    2586 generic.go:356] "Generic (PLEG): container finished" podID="09f4dac5-8796-44a4-877d-ecc640c815dc" containerID="7ccf914edd0d3d1f95bd2ca8328642e150d7d9bcd50ceae66387295389db8177" exitCode=0
Sep 17 18:46:47.269384 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.269331    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg" event={"ID":"09f4dac5-8796-44a4-877d-ecc640c815dc","Type":"ContainerDied","Data":"7ccf914edd0d3d1f95bd2ca8328642e150d7d9bcd50ceae66387295389db8177"}
Sep 17 18:46:47.269937 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.269397    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg" event={"ID":"09f4dac5-8796-44a4-877d-ecc640c815dc","Type":"ContainerStarted","Data":"1e9ccd47a1763f2688cf9d4c3655f31c31aa1a72cea7d0bfb13f59a032fa5a20"}
Sep 17 18:46:47.271053 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.270997    2586 generic.go:356] "Generic (PLEG): container finished" podID="a33b9774-3193-418f-b8d9-e65403df617a" containerID="3cc73e82e8346c3288a3e7dd753697e8c52ff7f6fc30bf32d12e3d4b27e252b8" exitCode=0
Sep 17 18:46:47.271053 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.271028    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6" event={"ID":"a33b9774-3193-418f-b8d9-e65403df617a","Type":"ContainerDied","Data":"3cc73e82e8346c3288a3e7dd753697e8c52ff7f6fc30bf32d12e3d4b27e252b8"}
Sep 17 18:46:47.271163 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.271069    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6" event={"ID":"a33b9774-3193-418f-b8d9-e65403df617a","Type":"ContainerStarted","Data":"cdb02a69fa1fe41c4d774e1534ad1ae0f812c2a4d4bc2f12326f2b0ef4f99418"}
Sep 17 18:46:47.272683 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.272662    2586 generic.go:356] "Generic (PLEG): container finished" podID="99443a86-943e-4b69-9867-df6117bb33fa" containerID="d78bb642a092f262109948dcb7567325a4dca6c92661a976a889e7644f6ff9f7" exitCode=0
Sep 17 18:46:47.272760 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.272745    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn" event={"ID":"99443a86-943e-4b69-9867-df6117bb33fa","Type":"ContainerDied","Data":"d78bb642a092f262109948dcb7567325a4dca6c92661a976a889e7644f6ff9f7"}
Sep 17 18:46:47.272808 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.272788    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn" event={"ID":"99443a86-943e-4b69-9867-df6117bb33fa","Type":"ContainerStarted","Data":"ccb5b4dda838abbde36efbcccf5a115140d63a0528e352d2ccdca12c344c21e7"}
Sep 17 18:46:47.274502 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.274481    2586 generic.go:356] "Generic (PLEG): container finished" podID="df2701f0-3c8b-43ef-ac58-11550fb6618e" containerID="1487c14086bca58261e17faf5266ea9ca60ca478f5cb8652da1ef91b9f9ab8b8" exitCode=0
Sep 17 18:46:47.274584 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.274515    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44" event={"ID":"df2701f0-3c8b-43ef-ac58-11550fb6618e","Type":"ContainerDied","Data":"1487c14086bca58261e17faf5266ea9ca60ca478f5cb8652da1ef91b9f9ab8b8"}
Sep 17 18:46:47.274584 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:47.274572    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44" event={"ID":"df2701f0-3c8b-43ef-ac58-11550fb6618e","Type":"ContainerStarted","Data":"71f562b132537f087f07c3824cd6bb5f70d94db82300a06555cddcfadcf4b970"}
Sep 17 18:46:48.281845 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:48.281795    2586 generic.go:356] "Generic (PLEG): container finished" podID="99443a86-943e-4b69-9867-df6117bb33fa" containerID="234192450cba4cd4e9a11a4e0c1d67b1e14ab0f990b9e162c4e82d3f2a12ce6b" exitCode=0
Sep 17 18:46:48.282195 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:48.281954    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn" event={"ID":"99443a86-943e-4b69-9867-df6117bb33fa","Type":"ContainerDied","Data":"234192450cba4cd4e9a11a4e0c1d67b1e14ab0f990b9e162c4e82d3f2a12ce6b"}
Sep 17 18:46:48.285287 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:48.284923    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg" event={"ID":"09f4dac5-8796-44a4-877d-ecc640c815dc","Type":"ContainerStarted","Data":"99a7083a3414a3f74ec06525675bc5ca464106c3531c3db72b9a3a1fa68941ec"}
Sep 17 18:46:49.292563 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:49.292523    2586 generic.go:356] "Generic (PLEG): container finished" podID="09f4dac5-8796-44a4-877d-ecc640c815dc" containerID="99a7083a3414a3f74ec06525675bc5ca464106c3531c3db72b9a3a1fa68941ec" exitCode=0
Sep 17 18:46:49.292563 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:49.292554    2586 generic.go:356] "Generic (PLEG): container finished" podID="09f4dac5-8796-44a4-877d-ecc640c815dc" containerID="d886249c5f7c48a5f9d14b61b7d894ccadeefc6bc6539eb419cb72c4f318c09f" exitCode=0
Sep 17 18:46:49.293115 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:49.292598    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg" event={"ID":"09f4dac5-8796-44a4-877d-ecc640c815dc","Type":"ContainerDied","Data":"99a7083a3414a3f74ec06525675bc5ca464106c3531c3db72b9a3a1fa68941ec"}
Sep 17 18:46:49.293115 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:49.292653    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg" event={"ID":"09f4dac5-8796-44a4-877d-ecc640c815dc","Type":"ContainerDied","Data":"d886249c5f7c48a5f9d14b61b7d894ccadeefc6bc6539eb419cb72c4f318c09f"}
Sep 17 18:46:49.294675 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:49.294649    2586 generic.go:356] "Generic (PLEG): container finished" podID="a33b9774-3193-418f-b8d9-e65403df617a" containerID="827d002139b158ed6947e6e881f421c158c5ecdb1e8b417166a897aa0041d439" exitCode=0
Sep 17 18:46:49.294792 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:49.294736    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6" event={"ID":"a33b9774-3193-418f-b8d9-e65403df617a","Type":"ContainerDied","Data":"827d002139b158ed6947e6e881f421c158c5ecdb1e8b417166a897aa0041d439"}
Sep 17 18:46:49.297114 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:49.297089    2586 generic.go:356] "Generic (PLEG): container finished" podID="99443a86-943e-4b69-9867-df6117bb33fa" containerID="a183db4a34b0427e3336af8e9c1edd3642526e59aa686361fc3eca35dabfe500" exitCode=0
Sep 17 18:46:49.297207 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:49.297113    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn" event={"ID":"99443a86-943e-4b69-9867-df6117bb33fa","Type":"ContainerDied","Data":"a183db4a34b0427e3336af8e9c1edd3642526e59aa686361fc3eca35dabfe500"}
Sep 17 18:46:50.304168 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.304080    2586 generic.go:356] "Generic (PLEG): container finished" podID="a33b9774-3193-418f-b8d9-e65403df617a" containerID="0bca2972450871afd275ebb22f9381fdae65d37d7f4a150dfdd1c82b9559eb40" exitCode=0
Sep 17 18:46:50.304556 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.304177    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6" event={"ID":"a33b9774-3193-418f-b8d9-e65403df617a","Type":"ContainerDied","Data":"0bca2972450871afd275ebb22f9381fdae65d37d7f4a150dfdd1c82b9559eb40"}
Sep 17 18:46:50.455234 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.455209    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:50.457922 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.457902    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:50.579165 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.579130    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-util\") pod \"99443a86-943e-4b69-9867-df6117bb33fa\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") "
Sep 17 18:46:50.579351 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.579177    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-util\") pod \"09f4dac5-8796-44a4-877d-ecc640c815dc\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") "
Sep 17 18:46:50.579351 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.579211    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dxbr\" (UniqueName: \"kubernetes.io/projected/09f4dac5-8796-44a4-877d-ecc640c815dc-kube-api-access-2dxbr\") pod \"09f4dac5-8796-44a4-877d-ecc640c815dc\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") "
Sep 17 18:46:50.579351 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.579280    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-bundle\") pod \"99443a86-943e-4b69-9867-df6117bb33fa\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") "
Sep 17 18:46:50.579351 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.579304    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-bundle\") pod \"09f4dac5-8796-44a4-877d-ecc640c815dc\" (UID: \"09f4dac5-8796-44a4-877d-ecc640c815dc\") "
Sep 17 18:46:50.579570 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.579369    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx5ls\" (UniqueName: \"kubernetes.io/projected/99443a86-943e-4b69-9867-df6117bb33fa-kube-api-access-mx5ls\") pod \"99443a86-943e-4b69-9867-df6117bb33fa\" (UID: \"99443a86-943e-4b69-9867-df6117bb33fa\") "
Sep 17 18:46:50.579899 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.579817    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-bundle" (OuterVolumeSpecName: "bundle") pod "99443a86-943e-4b69-9867-df6117bb33fa" (UID: "99443a86-943e-4b69-9867-df6117bb33fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:50.580335 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.580300    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-bundle" (OuterVolumeSpecName: "bundle") pod "09f4dac5-8796-44a4-877d-ecc640c815dc" (UID: "09f4dac5-8796-44a4-877d-ecc640c815dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:50.581635 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.581605    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f4dac5-8796-44a4-877d-ecc640c815dc-kube-api-access-2dxbr" (OuterVolumeSpecName: "kube-api-access-2dxbr") pod "09f4dac5-8796-44a4-877d-ecc640c815dc" (UID: "09f4dac5-8796-44a4-877d-ecc640c815dc"). InnerVolumeSpecName "kube-api-access-2dxbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:50.581722 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.581706    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99443a86-943e-4b69-9867-df6117bb33fa-kube-api-access-mx5ls" (OuterVolumeSpecName: "kube-api-access-mx5ls") pod "99443a86-943e-4b69-9867-df6117bb33fa" (UID: "99443a86-943e-4b69-9867-df6117bb33fa"). InnerVolumeSpecName "kube-api-access-mx5ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:50.585447 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.585427    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-util" (OuterVolumeSpecName: "util") pod "99443a86-943e-4b69-9867-df6117bb33fa" (UID: "99443a86-943e-4b69-9867-df6117bb33fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:50.587756 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.587734    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-util" (OuterVolumeSpecName: "util") pod "09f4dac5-8796-44a4-877d-ecc640c815dc" (UID: "09f4dac5-8796-44a4-877d-ecc640c815dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:50.680595 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.680559    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:50.680595 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.680586    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:50.680595 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.680596    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx5ls\" (UniqueName: \"kubernetes.io/projected/99443a86-943e-4b69-9867-df6117bb33fa-kube-api-access-mx5ls\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:50.680595 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.680604    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99443a86-943e-4b69-9867-df6117bb33fa-util\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:50.680595 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.680611    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09f4dac5-8796-44a4-877d-ecc640c815dc-util\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:50.680909 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:50.680618    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dxbr\" (UniqueName: \"kubernetes.io/projected/09f4dac5-8796-44a4-877d-ecc640c815dc-kube-api-access-2dxbr\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:51.311067 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.311039    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn"
Sep 17 18:46:51.311499 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.311078    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mdgdn" event={"ID":"99443a86-943e-4b69-9867-df6117bb33fa","Type":"ContainerDied","Data":"ccb5b4dda838abbde36efbcccf5a115140d63a0528e352d2ccdca12c344c21e7"}
Sep 17 18:46:51.311499 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.311130    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb5b4dda838abbde36efbcccf5a115140d63a0528e352d2ccdca12c344c21e7"
Sep 17 18:46:51.313121 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.313097    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg" event={"ID":"09f4dac5-8796-44a4-877d-ecc640c815dc","Type":"ContainerDied","Data":"1e9ccd47a1763f2688cf9d4c3655f31c31aa1a72cea7d0bfb13f59a032fa5a20"}
Sep 17 18:46:51.313219 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.313137    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e9ccd47a1763f2688cf9d4c3655f31c31aa1a72cea7d0bfb13f59a032fa5a20"
Sep 17 18:46:51.313219 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.313138    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac6bhg"
Sep 17 18:46:51.435142 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.435119    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:51.486730 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.486690    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5lkk\" (UniqueName: \"kubernetes.io/projected/a33b9774-3193-418f-b8d9-e65403df617a-kube-api-access-b5lkk\") pod \"a33b9774-3193-418f-b8d9-e65403df617a\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") "
Sep 17 18:46:51.486936 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.486762    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-bundle\") pod \"a33b9774-3193-418f-b8d9-e65403df617a\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") "
Sep 17 18:46:51.486936 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.486821    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-util\") pod \"a33b9774-3193-418f-b8d9-e65403df617a\" (UID: \"a33b9774-3193-418f-b8d9-e65403df617a\") "
Sep 17 18:46:51.487684 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.487485    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-bundle" (OuterVolumeSpecName: "bundle") pod "a33b9774-3193-418f-b8d9-e65403df617a" (UID: "a33b9774-3193-418f-b8d9-e65403df617a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:51.489474 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.489452    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33b9774-3193-418f-b8d9-e65403df617a-kube-api-access-b5lkk" (OuterVolumeSpecName: "kube-api-access-b5lkk") pod "a33b9774-3193-418f-b8d9-e65403df617a" (UID: "a33b9774-3193-418f-b8d9-e65403df617a"). InnerVolumeSpecName "kube-api-access-b5lkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:51.493284 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.493256    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-util" (OuterVolumeSpecName: "util") pod "a33b9774-3193-418f-b8d9-e65403df617a" (UID: "a33b9774-3193-418f-b8d9-e65403df617a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:51.588493 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.588449    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-util\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:51.588493 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.588494    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lkk\" (UniqueName: \"kubernetes.io/projected/a33b9774-3193-418f-b8d9-e65403df617a-kube-api-access-b5lkk\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:51.588681 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:51.588510    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33b9774-3193-418f-b8d9-e65403df617a-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:46:52.321227 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:52.321191    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6" event={"ID":"a33b9774-3193-418f-b8d9-e65403df617a","Type":"ContainerDied","Data":"cdb02a69fa1fe41c4d774e1534ad1ae0f812c2a4d4bc2f12326f2b0ef4f99418"}
Sep 17 18:46:52.321603 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:52.321238    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpppn6"
Sep 17 18:46:52.321603 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:46:52.321264    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdb02a69fa1fe41c4d774e1534ad1ae0f812c2a4d4bc2f12326f2b0ef4f99418"
Sep 17 18:47:01.373033 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:01.373000    2586 generic.go:356] "Generic (PLEG): container finished" podID="df2701f0-3c8b-43ef-ac58-11550fb6618e" containerID="999b66810bed74ec206736b50d98e8b0bdcaac7a728af0d8daf18e2578ec5c04" exitCode=0
Sep 17 18:47:01.373394 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:01.373088    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44" event={"ID":"df2701f0-3c8b-43ef-ac58-11550fb6618e","Type":"ContainerDied","Data":"999b66810bed74ec206736b50d98e8b0bdcaac7a728af0d8daf18e2578ec5c04"}
Sep 17 18:47:02.380891 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:02.380854    2586 generic.go:356] "Generic (PLEG): container finished" podID="df2701f0-3c8b-43ef-ac58-11550fb6618e" containerID="7bbd76f7fd345709f630852ee426548e84a213d4d82c060c2d62a62fb09e9377" exitCode=0
Sep 17 18:47:02.381327 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:02.380955    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44" event={"ID":"df2701f0-3c8b-43ef-ac58-11550fb6618e","Type":"ContainerDied","Data":"7bbd76f7fd345709f630852ee426548e84a213d4d82c060c2d62a62fb09e9377"}
Sep 17 18:47:03.512149 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:03.512124    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:47:03.586857 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:03.586792    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-util\") pod \"df2701f0-3c8b-43ef-ac58-11550fb6618e\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") "
Sep 17 18:47:03.586857 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:03.586860    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-bundle\") pod \"df2701f0-3c8b-43ef-ac58-11550fb6618e\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") "
Sep 17 18:47:03.587077 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:03.586891    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftl2r\" (UniqueName: \"kubernetes.io/projected/df2701f0-3c8b-43ef-ac58-11550fb6618e-kube-api-access-ftl2r\") pod \"df2701f0-3c8b-43ef-ac58-11550fb6618e\" (UID: \"df2701f0-3c8b-43ef-ac58-11550fb6618e\") "
Sep 17 18:47:03.587334 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:03.587311    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-bundle" (OuterVolumeSpecName: "bundle") pod "df2701f0-3c8b-43ef-ac58-11550fb6618e" (UID: "df2701f0-3c8b-43ef-ac58-11550fb6618e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:03.589349 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:03.589319    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2701f0-3c8b-43ef-ac58-11550fb6618e-kube-api-access-ftl2r" (OuterVolumeSpecName: "kube-api-access-ftl2r") pod "df2701f0-3c8b-43ef-ac58-11550fb6618e" (UID: "df2701f0-3c8b-43ef-ac58-11550fb6618e"). InnerVolumeSpecName "kube-api-access-ftl2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:03.591260 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:03.591240    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-util" (OuterVolumeSpecName: "util") pod "df2701f0-3c8b-43ef-ac58-11550fb6618e" (UID: "df2701f0-3c8b-43ef-ac58-11550fb6618e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:03.688123 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:03.688011    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-util\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:47:03.688123 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:03.688064    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df2701f0-3c8b-43ef-ac58-11550fb6618e-bundle\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:47:03.688123 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:03.688079    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftl2r\" (UniqueName: \"kubernetes.io/projected/df2701f0-3c8b-43ef-ac58-11550fb6618e-kube-api-access-ftl2r\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:47:04.391535 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:04.391499    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44" event={"ID":"df2701f0-3c8b-43ef-ac58-11550fb6618e","Type":"ContainerDied","Data":"71f562b132537f087f07c3824cd6bb5f70d94db82300a06555cddcfadcf4b970"}
Sep 17 18:47:04.391535 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:04.391526    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5c44"
Sep 17 18:47:04.391751 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:04.391546    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f562b132537f087f07c3824cd6bb5f70d94db82300a06555cddcfadcf4b970"
Sep 17 18:47:09.323006 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.322974    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5"]
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323292    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33b9774-3193-418f-b8d9-e65403df617a" containerName="util"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323303    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33b9774-3193-418f-b8d9-e65403df617a" containerName="util"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323315    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2701f0-3c8b-43ef-ac58-11550fb6618e" containerName="util"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323322    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2701f0-3c8b-43ef-ac58-11550fb6618e" containerName="util"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323334    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99443a86-943e-4b69-9867-df6117bb33fa" containerName="extract"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323339    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="99443a86-943e-4b69-9867-df6117bb33fa" containerName="extract"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323347    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f4dac5-8796-44a4-877d-ecc640c815dc" containerName="util"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323353    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f4dac5-8796-44a4-877d-ecc640c815dc" containerName="util"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323358    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33b9774-3193-418f-b8d9-e65403df617a" containerName="pull"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323363    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33b9774-3193-418f-b8d9-e65403df617a" containerName="pull"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323368    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2701f0-3c8b-43ef-ac58-11550fb6618e" containerName="extract"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323373    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2701f0-3c8b-43ef-ac58-11550fb6618e" containerName="extract"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323385    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99443a86-943e-4b69-9867-df6117bb33fa" containerName="pull"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323390    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="99443a86-943e-4b69-9867-df6117bb33fa" containerName="pull"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323395    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f4dac5-8796-44a4-877d-ecc640c815dc" containerName="pull"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323399    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f4dac5-8796-44a4-877d-ecc640c815dc" containerName="pull"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323404    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2701f0-3c8b-43ef-ac58-11550fb6618e" containerName="pull"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323409    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2701f0-3c8b-43ef-ac58-11550fb6618e" containerName="pull"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323414    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33b9774-3193-418f-b8d9-e65403df617a" containerName="extract"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323420    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33b9774-3193-418f-b8d9-e65403df617a" containerName="extract"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323426    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f4dac5-8796-44a4-877d-ecc640c815dc" containerName="extract"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323432    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f4dac5-8796-44a4-877d-ecc640c815dc" containerName="extract"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323438    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99443a86-943e-4b69-9867-df6117bb33fa" containerName="util"
Sep 17 18:47:09.323465 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323442    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="99443a86-943e-4b69-9867-df6117bb33fa" containerName="util"
Sep 17 18:47:09.324191 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323492    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33b9774-3193-418f-b8d9-e65403df617a" containerName="extract"
Sep 17 18:47:09.324191 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323501    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="09f4dac5-8796-44a4-877d-ecc640c815dc" containerName="extract"
Sep 17 18:47:09.324191 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323507    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="99443a86-943e-4b69-9867-df6117bb33fa" containerName="extract"
Sep 17 18:47:09.324191 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.323514    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="df2701f0-3c8b-43ef-ac58-11550fb6618e" containerName="extract"
Sep 17 18:47:09.325688 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.325673    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5"
Sep 17 18:47:09.328592 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.328569    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 18:47:09.328733 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.328673    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lnqpp\""
Sep 17 18:47:09.329471 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.329455    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 18:47:09.344851 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.342289    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5"]
Sep 17 18:47:09.440360 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.440320    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-698n6\" (UniqueName: \"kubernetes.io/projected/cbb4a599-0b6e-450d-903b-f7f5b8780f17-kube-api-access-698n6\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-2nph5\" (UID: \"cbb4a599-0b6e-450d-903b-f7f5b8780f17\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5"
Sep 17 18:47:09.541564 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.541526    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-698n6\" (UniqueName: \"kubernetes.io/projected/cbb4a599-0b6e-450d-903b-f7f5b8780f17-kube-api-access-698n6\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-2nph5\" (UID: \"cbb4a599-0b6e-450d-903b-f7f5b8780f17\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5"
Sep 17 18:47:09.556623 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.556583    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-698n6\" (UniqueName: \"kubernetes.io/projected/cbb4a599-0b6e-450d-903b-f7f5b8780f17-kube-api-access-698n6\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-2nph5\" (UID: \"cbb4a599-0b6e-450d-903b-f7f5b8780f17\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5"
Sep 17 18:47:09.636011 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.635921    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5"
Sep 17 18:47:09.914136 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:09.914107    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5"]
Sep 17 18:47:09.915877 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:47:09.915820    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb4a599_0b6e_450d_903b_f7f5b8780f17.slice/crio-966177f5b3cec45dce34b8607360ffe00eb0ad351cd7fe503bf3c22b96d6efa4 WatchSource:0}: Error finding container 966177f5b3cec45dce34b8607360ffe00eb0ad351cd7fe503bf3c22b96d6efa4: Status 404 returned error can't find the container with id 966177f5b3cec45dce34b8607360ffe00eb0ad351cd7fe503bf3c22b96d6efa4
Sep 17 18:47:10.422038 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:10.422004    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5" event={"ID":"cbb4a599-0b6e-450d-903b-f7f5b8780f17","Type":"ContainerStarted","Data":"966177f5b3cec45dce34b8607360ffe00eb0ad351cd7fe503bf3c22b96d6efa4"}
Sep 17 18:47:11.480529 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:11.480502    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"]
Sep 17 18:47:11.483849 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:11.483782    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"
Sep 17 18:47:11.486381 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:47:11.486353    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"dns-operator-controller-manager-65dc8f76f5-h85px\" is forbidden: User \"system:node:ip-10-0-129-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-204.ec2.internal' and this object" podUID="0bf2898d-6df1-4592-a611-d0d420ff70c6" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"
Sep 17 18:47:11.486654 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:47:11.486623    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-operator-controller-manager-dockercfg-x9fpp\" is forbidden: User \"system:node:ip-10-0-129-204.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-204.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-x9fpp\"" type="*v1.Secret"
Sep 17 18:47:11.487616 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:47:11.487593    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-operator-controller-env\" is forbidden: User \"system:node:ip-10-0-129-204.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-204.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" type="*v1.ConfigMap"
Sep 17 18:47:11.508164 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:11.508135    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"]
Sep 17 18:47:11.663202 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:11.663160    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btt6r\" (UniqueName: \"kubernetes.io/projected/0bf2898d-6df1-4592-a611-d0d420ff70c6-kube-api-access-btt6r\") pod \"dns-operator-controller-manager-65dc8f76f5-h85px\" (UID: \"0bf2898d-6df1-4592-a611-d0d420ff70c6\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"
Sep 17 18:47:11.763895 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:11.763805    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btt6r\" (UniqueName: \"kubernetes.io/projected/0bf2898d-6df1-4592-a611-d0d420ff70c6-kube-api-access-btt6r\") pod \"dns-operator-controller-manager-65dc8f76f5-h85px\" (UID: \"0bf2898d-6df1-4592-a611-d0d420ff70c6\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"
Sep 17 18:47:11.799490 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:11.799458    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btt6r\" (UniqueName: \"kubernetes.io/projected/0bf2898d-6df1-4592-a611-d0d420ff70c6-kube-api-access-btt6r\") pod \"dns-operator-controller-manager-65dc8f76f5-h85px\" (UID: \"0bf2898d-6df1-4592-a611-d0d420ff70c6\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"
Sep 17 18:47:12.347864 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:12.347814    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 18:47:12.432641 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:12.432606    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5" event={"ID":"cbb4a599-0b6e-450d-903b-f7f5b8780f17","Type":"ContainerStarted","Data":"0b101cf22ebc83079e3ddbcbde25c49e8239114a9b7649a609b5407bca5efc70"}
Sep 17 18:47:12.432819 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:12.432667    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5"
Sep 17 18:47:12.469016 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:12.468985    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-x9fpp\""
Sep 17 18:47:12.475802 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:12.475781    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"
Sep 17 18:47:12.509777 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:12.509722    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5" podStartSLOduration=1.986518631 podStartE2EDuration="3.509705063s" totalImagesPullingTime="1.523186432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:12.469121051 +0000 UTC m=+453.328945718" watchObservedRunningTime="2026-09-17 18:47:12.509705063 +0000 UTC m=+453.369529753"
Sep 17 18:47:12.767069 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:12.766968    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"]
Sep 17 18:47:12.769590 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:47:12.769548    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf2898d_6df1_4592_a611_d0d420ff70c6.slice/crio-ea82f397ce8d4b51c0e4615f1ecaf5f4043f87002e7b45e719c9f3e7d72e32f8 WatchSource:0}: Error finding container ea82f397ce8d4b51c0e4615f1ecaf5f4043f87002e7b45e719c9f3e7d72e32f8: Status 404 returned error can't find the container with id ea82f397ce8d4b51c0e4615f1ecaf5f4043f87002e7b45e719c9f3e7d72e32f8
Sep 17 18:47:13.439462 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:13.439428    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px" event={"ID":"0bf2898d-6df1-4592-a611-d0d420ff70c6","Type":"ContainerStarted","Data":"ea82f397ce8d4b51c0e4615f1ecaf5f4043f87002e7b45e719c9f3e7d72e32f8"}
Sep 17 18:47:14.560891 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:14.560866    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-bhdd9"]
Sep 17 18:47:14.563386 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:14.563372    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-bhdd9"
Sep 17 18:47:14.568061 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:14.568043    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8npb5\""
Sep 17 18:47:14.585648 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:14.585622    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-bhdd9"]
Sep 17 18:47:14.689416 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:14.689385    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4grc\" (UniqueName: \"kubernetes.io/projected/f30db5da-b425-4dd7-8ceb-18e9b10ce350-kube-api-access-p4grc\") pod \"authorino-operator-6fc8f864f4-bhdd9\" (UID: \"f30db5da-b425-4dd7-8ceb-18e9b10ce350\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-bhdd9"
Sep 17 18:47:14.790414 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:14.790368    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4grc\" (UniqueName: \"kubernetes.io/projected/f30db5da-b425-4dd7-8ceb-18e9b10ce350-kube-api-access-p4grc\") pod \"authorino-operator-6fc8f864f4-bhdd9\" (UID: \"f30db5da-b425-4dd7-8ceb-18e9b10ce350\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-bhdd9"
Sep 17 18:47:14.799765 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:14.799730    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4grc\" (UniqueName: \"kubernetes.io/projected/f30db5da-b425-4dd7-8ceb-18e9b10ce350-kube-api-access-p4grc\") pod \"authorino-operator-6fc8f864f4-bhdd9\" (UID: \"f30db5da-b425-4dd7-8ceb-18e9b10ce350\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-bhdd9"
Sep 17 18:47:14.950747 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:14.950716    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-bhdd9"
Sep 17 18:47:15.232513 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:15.232482    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-bhdd9"]
Sep 17 18:47:15.236568 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:47:15.236536    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30db5da_b425_4dd7_8ceb_18e9b10ce350.slice/crio-ff3dbfa1ff7284ba1998bb4e5d85830b72edde755944064f3d0e268b9af813ce WatchSource:0}: Error finding container ff3dbfa1ff7284ba1998bb4e5d85830b72edde755944064f3d0e268b9af813ce: Status 404 returned error can't find the container with id ff3dbfa1ff7284ba1998bb4e5d85830b72edde755944064f3d0e268b9af813ce
Sep 17 18:47:15.457485 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:15.457455    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"
Sep 17 18:47:15.457648 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:15.457503    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-bhdd9" event={"ID":"f30db5da-b425-4dd7-8ceb-18e9b10ce350","Type":"ContainerStarted","Data":"ff3dbfa1ff7284ba1998bb4e5d85830b72edde755944064f3d0e268b9af813ce"}
Sep 17 18:47:15.457648 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:15.457526    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px" event={"ID":"0bf2898d-6df1-4592-a611-d0d420ff70c6","Type":"ContainerStarted","Data":"77d9f77da8e19cfcfa613acfc54e13317b97cc753221f48af67f02d8486ffe8f"}
Sep 17 18:47:15.483002 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:15.482902    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px" podStartSLOduration=2.701528848 podStartE2EDuration="4.482885136s" totalImagesPullingTime="1.781356288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:15.480580424 +0000 UTC m=+456.340405089" watchObservedRunningTime="2026-09-17 18:47:15.482885136 +0000 UTC m=+456.342709784"
Sep 17 18:47:17.468053 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:17.468017    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-bhdd9" event={"ID":"f30db5da-b425-4dd7-8ceb-18e9b10ce350","Type":"ContainerStarted","Data":"c14b94e50c779d7ce8ffa14f5c2670ab5c1c0085cd6e08295bfdd32e334dbcc7"}
Sep 17 18:47:17.468463 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:17.468078    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-bhdd9"
Sep 17 18:47:17.491776 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:17.491721    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-bhdd9" podStartSLOduration=1.990084672 podStartE2EDuration="3.491703292s" totalImagesPullingTime="1.50161862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:17.490115468 +0000 UTC m=+458.349940129" watchObservedRunningTime="2026-09-17 18:47:17.491703292 +0000 UTC m=+458.351527942"
Sep 17 18:47:23.442519 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:23.442485    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-2nph5"
Sep 17 18:47:26.463624 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.463593    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-h85px"
Sep 17 18:47:26.575643 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.575604    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"]
Sep 17 18:47:26.578429 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.578412    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:26.583475 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.583452    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 18:47:26.584241 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.584220    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 18:47:26.584791 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.584771    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wm6w4\""
Sep 17 18:47:26.589124 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.589099    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"]
Sep 17 18:47:26.591815 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.591787    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5b293af-3321-46ca-8444-b93ed864838c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5sqq4\" (UID: \"a5b293af-3321-46ca-8444-b93ed864838c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:26.591928 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.591859    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5b293af-3321-46ca-8444-b93ed864838c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5sqq4\" (UID: \"a5b293af-3321-46ca-8444-b93ed864838c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:26.592002 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.591933    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srwm\" (UniqueName: \"kubernetes.io/projected/a5b293af-3321-46ca-8444-b93ed864838c-kube-api-access-8srwm\") pod \"kuadrant-console-plugin-fbdd4f445-5sqq4\" (UID: \"a5b293af-3321-46ca-8444-b93ed864838c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:26.692703 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.692669    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srwm\" (UniqueName: \"kubernetes.io/projected/a5b293af-3321-46ca-8444-b93ed864838c-kube-api-access-8srwm\") pod \"kuadrant-console-plugin-fbdd4f445-5sqq4\" (UID: \"a5b293af-3321-46ca-8444-b93ed864838c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:26.692906 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.692710    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5b293af-3321-46ca-8444-b93ed864838c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5sqq4\" (UID: \"a5b293af-3321-46ca-8444-b93ed864838c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:26.692906 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.692754    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5b293af-3321-46ca-8444-b93ed864838c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5sqq4\" (UID: \"a5b293af-3321-46ca-8444-b93ed864838c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:26.693426 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.693407    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5b293af-3321-46ca-8444-b93ed864838c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5sqq4\" (UID: \"a5b293af-3321-46ca-8444-b93ed864838c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:26.695326 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.695305    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5b293af-3321-46ca-8444-b93ed864838c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5sqq4\" (UID: \"a5b293af-3321-46ca-8444-b93ed864838c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:26.703850 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.703810    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srwm\" (UniqueName: \"kubernetes.io/projected/a5b293af-3321-46ca-8444-b93ed864838c-kube-api-access-8srwm\") pod \"kuadrant-console-plugin-fbdd4f445-5sqq4\" (UID: \"a5b293af-3321-46ca-8444-b93ed864838c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:26.887691 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:26.887658    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"
Sep 17 18:47:27.173474 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:27.173404    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4"]
Sep 17 18:47:27.177203 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:47:27.177178    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5b293af_3321_46ca_8444_b93ed864838c.slice/crio-c4a0b0f967553225f199cc942f751c76588cb5d016f10ecd7d5bea0d42ebb442 WatchSource:0}: Error finding container c4a0b0f967553225f199cc942f751c76588cb5d016f10ecd7d5bea0d42ebb442: Status 404 returned error can't find the container with id c4a0b0f967553225f199cc942f751c76588cb5d016f10ecd7d5bea0d42ebb442
Sep 17 18:47:27.527271 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:27.527184    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4" event={"ID":"a5b293af-3321-46ca-8444-b93ed864838c","Type":"ContainerStarted","Data":"c4a0b0f967553225f199cc942f751c76588cb5d016f10ecd7d5bea0d42ebb442"}
Sep 17 18:47:28.476129 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:28.476095    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-bhdd9"
Sep 17 18:47:32.558774 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:32.558739    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4" event={"ID":"a5b293af-3321-46ca-8444-b93ed864838c","Type":"ContainerStarted","Data":"a5262ee0e7325bb072322482a4bf04b3fcb9fb61d1e469e057515d4da0d4c2b4"}
Sep 17 18:47:32.595870 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:47:32.595801    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5sqq4" podStartSLOduration=1.961335315 podStartE2EDuration="6.595784705s" totalImagesPullingTime="4.63444939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:32.593258454 +0000 UTC m=+473.453083102" watchObservedRunningTime="2026-09-17 18:47:32.595784705 +0000 UTC m=+473.455609353"
Sep 17 18:48:09.416925 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.416883    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tpmcb"]
Sep 17 18:48:09.435144 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.435113    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tpmcb"]
Sep 17 18:48:09.435297 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.435208    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:09.437668 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.437643    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 18:48:09.519265 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.519231    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tpmcb"]
Sep 17 18:48:09.569233 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.569187    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lv6\" (UniqueName: \"kubernetes.io/projected/bf8fa143-16ec-42a2-a51c-334025fa103c-kube-api-access-67lv6\") pod \"limitador-limitador-8588749977-tpmcb\" (UID: \"bf8fa143-16ec-42a2-a51c-334025fa103c\") " pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:09.569429 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.569392    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf8fa143-16ec-42a2-a51c-334025fa103c-config-file\") pod \"limitador-limitador-8588749977-tpmcb\" (UID: \"bf8fa143-16ec-42a2-a51c-334025fa103c\") " pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:09.670530 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.670438    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf8fa143-16ec-42a2-a51c-334025fa103c-config-file\") pod \"limitador-limitador-8588749977-tpmcb\" (UID: \"bf8fa143-16ec-42a2-a51c-334025fa103c\") " pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:09.670701 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.670552    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lv6\" (UniqueName: \"kubernetes.io/projected/bf8fa143-16ec-42a2-a51c-334025fa103c-kube-api-access-67lv6\") pod \"limitador-limitador-8588749977-tpmcb\" (UID: \"bf8fa143-16ec-42a2-a51c-334025fa103c\") " pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:09.671087 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.671068    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf8fa143-16ec-42a2-a51c-334025fa103c-config-file\") pod \"limitador-limitador-8588749977-tpmcb\" (UID: \"bf8fa143-16ec-42a2-a51c-334025fa103c\") " pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:09.680124 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.680097    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lv6\" (UniqueName: \"kubernetes.io/projected/bf8fa143-16ec-42a2-a51c-334025fa103c-kube-api-access-67lv6\") pod \"limitador-limitador-8588749977-tpmcb\" (UID: \"bf8fa143-16ec-42a2-a51c-334025fa103c\") " pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:09.745735 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:09.745701    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:10.027221 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:10.027197    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tpmcb"]
Sep 17 18:48:10.029451 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:48:10.029426    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8fa143_16ec_42a2_a51c_334025fa103c.slice/crio-d7b740e44a6bed872627c45292ba3b256bd68f4f39833f35deee471b2d22388b WatchSource:0}: Error finding container d7b740e44a6bed872627c45292ba3b256bd68f4f39833f35deee471b2d22388b: Status 404 returned error can't find the container with id d7b740e44a6bed872627c45292ba3b256bd68f4f39833f35deee471b2d22388b
Sep 17 18:48:10.751984 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:10.751945    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb" event={"ID":"bf8fa143-16ec-42a2-a51c-334025fa103c","Type":"ContainerStarted","Data":"d7b740e44a6bed872627c45292ba3b256bd68f4f39833f35deee471b2d22388b"}
Sep 17 18:48:11.758879 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:11.758765    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb" event={"ID":"bf8fa143-16ec-42a2-a51c-334025fa103c","Type":"ContainerStarted","Data":"e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76"}
Sep 17 18:48:11.759295 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:11.758973    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:11.777510 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:11.777459    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb" podStartSLOduration=1.491820259 podStartE2EDuration="2.777443792s" totalImagesPullingTime="1.285623533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:11.776175093 +0000 UTC m=+512.635999794" watchObservedRunningTime="2026-09-17 18:48:11.777443792 +0000 UTC m=+512.637268439"
Sep 17 18:48:22.765427 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:22.765346    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:25.617524 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:25.617484    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tpmcb"]
Sep 17 18:48:25.617926 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:25.617717    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb" podUID="bf8fa143-16ec-42a2-a51c-334025fa103c" containerName="limitador" containerID="cri-o://e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76" gracePeriod=30
Sep 17 18:48:26.161765 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.161740    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:26.201466 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.201384    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67lv6\" (UniqueName: \"kubernetes.io/projected/bf8fa143-16ec-42a2-a51c-334025fa103c-kube-api-access-67lv6\") pod \"bf8fa143-16ec-42a2-a51c-334025fa103c\" (UID: \"bf8fa143-16ec-42a2-a51c-334025fa103c\") "
Sep 17 18:48:26.201617 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.201480    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf8fa143-16ec-42a2-a51c-334025fa103c-config-file\") pod \"bf8fa143-16ec-42a2-a51c-334025fa103c\" (UID: \"bf8fa143-16ec-42a2-a51c-334025fa103c\") "
Sep 17 18:48:26.201860 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.201815    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8fa143-16ec-42a2-a51c-334025fa103c-config-file" (OuterVolumeSpecName: "config-file") pod "bf8fa143-16ec-42a2-a51c-334025fa103c" (UID: "bf8fa143-16ec-42a2-a51c-334025fa103c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:48:26.203733 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.203708    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8fa143-16ec-42a2-a51c-334025fa103c-kube-api-access-67lv6" (OuterVolumeSpecName: "kube-api-access-67lv6") pod "bf8fa143-16ec-42a2-a51c-334025fa103c" (UID: "bf8fa143-16ec-42a2-a51c-334025fa103c"). InnerVolumeSpecName "kube-api-access-67lv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:48:26.302736 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.302699    2586 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf8fa143-16ec-42a2-a51c-334025fa103c-config-file\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:48:26.302736 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.302732    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67lv6\" (UniqueName: \"kubernetes.io/projected/bf8fa143-16ec-42a2-a51c-334025fa103c-kube-api-access-67lv6\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:48:26.844570 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.844531    2586 generic.go:356] "Generic (PLEG): container finished" podID="bf8fa143-16ec-42a2-a51c-334025fa103c" containerID="e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76" exitCode=0
Sep 17 18:48:26.845058 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.844582    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb"
Sep 17 18:48:26.845058 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.844615    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb" event={"ID":"bf8fa143-16ec-42a2-a51c-334025fa103c","Type":"ContainerDied","Data":"e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76"}
Sep 17 18:48:26.845058 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.844669    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tpmcb" event={"ID":"bf8fa143-16ec-42a2-a51c-334025fa103c","Type":"ContainerDied","Data":"d7b740e44a6bed872627c45292ba3b256bd68f4f39833f35deee471b2d22388b"}
Sep 17 18:48:26.845058 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.844689    2586 scope.go:117] "RemoveContainer" containerID="e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76"
Sep 17 18:48:26.853940 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.853920    2586 scope.go:117] "RemoveContainer" containerID="e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76"
Sep 17 18:48:26.854209 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:48:26.854187    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76\": container with ID starting with e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76 not found: ID does not exist" containerID="e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76"
Sep 17 18:48:26.854271 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.854217    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76"} err="failed to get container status \"e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76\": rpc error: code = NotFound desc = could not find container \"e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76\": container with ID starting with e9028820ff406e3d4f4c1882aa1710c51117a4aecf1367bd01f7afa4429e6d76 not found: ID does not exist"
Sep 17 18:48:26.867389 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.867360    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tpmcb"]
Sep 17 18:48:26.870871 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:26.870848    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tpmcb"]
Sep 17 18:48:27.459150 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:27.459119    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf8fa143-16ec-42a2-a51c-334025fa103c" path="/var/lib/kubelet/pods/bf8fa143-16ec-42a2-a51c-334025fa103c/volumes"
Sep 17 18:48:57.865483 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.865452    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"]
Sep 17 18:48:57.865945 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.865788    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf8fa143-16ec-42a2-a51c-334025fa103c" containerName="limitador"
Sep 17 18:48:57.865945 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.865799    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf8fa143-16ec-42a2-a51c-334025fa103c" containerName="limitador"
Sep 17 18:48:57.865945 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.865907    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf8fa143-16ec-42a2-a51c-334025fa103c" containerName="limitador"
Sep 17 18:48:57.869070 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.869049    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:57.895147 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.895118    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"]
Sep 17 18:48:57.984021 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.983970    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4dc816e7-abf9-4437-b226-ec80b9378fec-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:57.984244 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.984042    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc816e7-abf9-4437-b226-ec80b9378fec-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:57.984244 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.984107    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:57.984244 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.984164    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:57.984244 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.984201    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:57.984482 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.984276    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k75js\" (UniqueName: \"kubernetes.io/projected/4dc816e7-abf9-4437-b226-ec80b9378fec-kube-api-access-k75js\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:57.984482 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:57.984339    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.085093 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.085054    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k75js\" (UniqueName: \"kubernetes.io/projected/4dc816e7-abf9-4437-b226-ec80b9378fec-kube-api-access-k75js\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.085271 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.085105    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.085271 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.085148    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4dc816e7-abf9-4437-b226-ec80b9378fec-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.085271 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.085177    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc816e7-abf9-4437-b226-ec80b9378fec-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.085271 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.085205    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.085271 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.085248    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.085519 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.085285    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.086329 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.086299    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.087900 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.087877    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc816e7-abf9-4437-b226-ec80b9378fec-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.087987 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.087921    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4dc816e7-abf9-4437-b226-ec80b9378fec-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.088038 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.087992    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.088038 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.088019    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.095130 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.095106    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc816e7-abf9-4437-b226-ec80b9378fec-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.102983 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.102963    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k75js\" (UniqueName: \"kubernetes.io/projected/4dc816e7-abf9-4437-b226-ec80b9378fec-kube-api-access-k75js\") pod \"istiod-openshift-gateway-5677ccb49b-f4n7q\" (UID: \"4dc816e7-abf9-4437-b226-ec80b9378fec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.178547 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.178453    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:58.491451 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.491362    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"]
Sep 17 18:48:58.498274 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:48:58.498247    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc816e7_abf9_4437_b226_ec80b9378fec.slice/crio-e20e691796d44bb800a1c1d88e5222a5b88bca07c809bd2fd2361cbaf5d20350 WatchSource:0}: Error finding container e20e691796d44bb800a1c1d88e5222a5b88bca07c809bd2fd2361cbaf5d20350: Status 404 returned error can't find the container with id e20e691796d44bb800a1c1d88e5222a5b88bca07c809bd2fd2361cbaf5d20350
Sep 17 18:48:58.500319 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.500274    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:48:58.500408 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:58.500342    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:48:59.021089 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:59.021042    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q" event={"ID":"4dc816e7-abf9-4437-b226-ec80b9378fec","Type":"ContainerStarted","Data":"2caa3235333249d0bc00f552488db097faaea2a9e4c88d10f2f4ccb984eee0e9"}
Sep 17 18:48:59.021581 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:59.021109    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q" event={"ID":"4dc816e7-abf9-4437-b226-ec80b9378fec","Type":"ContainerStarted","Data":"e20e691796d44bb800a1c1d88e5222a5b88bca07c809bd2fd2361cbaf5d20350"}
Sep 17 18:48:59.021581 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:59.021320    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:48:59.023690 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:59.023624    2586 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-f4n7q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 18:48:59.023690 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:59.023680    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q" podUID="4dc816e7-abf9-4437-b226-ec80b9378fec" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:48:59.093418 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:48:59.093351    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q" podStartSLOduration=2.093329325 podStartE2EDuration="2.093329325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:59.091397032 +0000 UTC m=+559.951221702" watchObservedRunningTime="2026-09-17 18:48:59.093329325 +0000 UTC m=+559.953153973"
Sep 17 18:49:00.027272 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.027240    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f4n7q"
Sep 17 18:49:00.174679 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.174648    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"]
Sep 17 18:49:00.174918 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.174863    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s" podUID="e9baf987-4ae7-42dc-8c0a-9249b2dee96f" containerName="discovery" containerID="cri-o://dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59" gracePeriod=30
Sep 17 18:49:00.430448 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.430420    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:49:00.608945 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.608901    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw26j\" (UniqueName: \"kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-kube-api-access-jw26j\") pod \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") "
Sep 17 18:49:00.609131 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.608960    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-kubeconfig\") pod \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") "
Sep 17 18:49:00.609131 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.608990    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-ca-configmap\") pod \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") "
Sep 17 18:49:00.609131 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.609020    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-local-certs\") pod \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") "
Sep 17 18:49:00.609131 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.609042    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-token\") pod \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") "
Sep 17 18:49:00.609131 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.609103    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-dns-cert\") pod \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") "
Sep 17 18:49:00.609394 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.609140    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-cacerts\") pod \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\" (UID: \"e9baf987-4ae7-42dc-8c0a-9249b2dee96f\") "
Sep 17 18:49:00.609484 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.609449    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e9baf987-4ae7-42dc-8c0a-9249b2dee96f" (UID: "e9baf987-4ae7-42dc-8c0a-9249b2dee96f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:49:00.611637 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.611598    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9baf987-4ae7-42dc-8c0a-9249b2dee96f" (UID: "e9baf987-4ae7-42dc-8c0a-9249b2dee96f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:00.611782 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.611664    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-kube-api-access-jw26j" (OuterVolumeSpecName: "kube-api-access-jw26j") pod "e9baf987-4ae7-42dc-8c0a-9249b2dee96f" (UID: "e9baf987-4ae7-42dc-8c0a-9249b2dee96f"). InnerVolumeSpecName "kube-api-access-jw26j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:00.611872 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.611807    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-local-certs" (OuterVolumeSpecName: "local-certs") pod "e9baf987-4ae7-42dc-8c0a-9249b2dee96f" (UID: "e9baf987-4ae7-42dc-8c0a-9249b2dee96f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:00.611945 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.611927    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e9baf987-4ae7-42dc-8c0a-9249b2dee96f" (UID: "e9baf987-4ae7-42dc-8c0a-9249b2dee96f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:00.612113 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.612087    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-cacerts" (OuterVolumeSpecName: "cacerts") pod "e9baf987-4ae7-42dc-8c0a-9249b2dee96f" (UID: "e9baf987-4ae7-42dc-8c0a-9249b2dee96f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:00.612113 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.612097    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e9baf987-4ae7-42dc-8c0a-9249b2dee96f" (UID: "e9baf987-4ae7-42dc-8c0a-9249b2dee96f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:00.709984 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.709946    2586 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-dns-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:49:00.709984 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.709974    2586 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-cacerts\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:49:00.709984 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.709985    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw26j\" (UniqueName: \"kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-kube-api-access-jw26j\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:49:00.709984 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.709996    2586 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-kubeconfig\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:49:00.710249 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.710004    2586 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-csr-ca-configmap\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:49:00.710249 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.710012    2586 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-local-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:49:00.710249 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:00.710020    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9baf987-4ae7-42dc-8c0a-9249b2dee96f-istio-token\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:49:01.032639 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:01.032548    2586 generic.go:356] "Generic (PLEG): container finished" podID="e9baf987-4ae7-42dc-8c0a-9249b2dee96f" containerID="dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59" exitCode=0
Sep 17 18:49:01.032639 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:01.032592    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"
Sep 17 18:49:01.033128 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:01.032635    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s" event={"ID":"e9baf987-4ae7-42dc-8c0a-9249b2dee96f","Type":"ContainerDied","Data":"dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59"}
Sep 17 18:49:01.033128 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:01.032683    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s" event={"ID":"e9baf987-4ae7-42dc-8c0a-9249b2dee96f","Type":"ContainerDied","Data":"a2245d0ae464b6dc4cd33a89f266cf36798fc895b8b5ac4680e4a39f97242cc4"}
Sep 17 18:49:01.033128 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:01.032703    2586 scope.go:117] "RemoveContainer" containerID="dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59"
Sep 17 18:49:01.042185 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:01.042170    2586 scope.go:117] "RemoveContainer" containerID="dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59"
Sep 17 18:49:01.042446 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:49:01.042429    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59\": container with ID starting with dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59 not found: ID does not exist" containerID="dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59"
Sep 17 18:49:01.042491 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:01.042455    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59"} err="failed to get container status \"dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59\": rpc error: code = NotFound desc = could not find container \"dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59\": container with ID starting with dd4d7330dc3dc54381e65dd65208b24e9147de4aa21a6baf50ea8788068f8a59 not found: ID does not exist"
Sep 17 18:49:01.056530 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:01.056497    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"]
Sep 17 18:49:01.059964 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:01.059940    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zw26s"]
Sep 17 18:49:01.459387 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:01.459355    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9baf987-4ae7-42dc-8c0a-9249b2dee96f" path="/var/lib/kubelet/pods/e9baf987-4ae7-42dc-8c0a-9249b2dee96f/volumes"
Sep 17 18:49:12.852065 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.852024    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7dcf484667-jgqhq"]
Sep 17 18:49:12.852658 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.852535    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9baf987-4ae7-42dc-8c0a-9249b2dee96f" containerName="discovery"
Sep 17 18:49:12.852658 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.852554    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9baf987-4ae7-42dc-8c0a-9249b2dee96f" containerName="discovery"
Sep 17 18:49:12.852658 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.852654    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9baf987-4ae7-42dc-8c0a-9249b2dee96f" containerName="discovery"
Sep 17 18:49:12.857223 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.857203    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:12.860169 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.860150    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 18:49:12.860931 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.860903    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:49:12.861023 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.860936    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-vhq2n\""
Sep 17 18:49:12.861023 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.860936    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:49:12.869846 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.869805    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf484667-jgqhq"]
Sep 17 18:49:12.900432 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.900400    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvpt6\" (UniqueName: \"kubernetes.io/projected/7df9edab-0c07-444d-9c7b-683d6599929a-kube-api-access-fvpt6\") pod \"kserve-controller-manager-7dcf484667-jgqhq\" (UID: \"7df9edab-0c07-444d-9c7b-683d6599929a\") " pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:12.900606 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.900468    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7df9edab-0c07-444d-9c7b-683d6599929a-cert\") pod \"kserve-controller-manager-7dcf484667-jgqhq\" (UID: \"7df9edab-0c07-444d-9c7b-683d6599929a\") " pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:12.915159 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.915129    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-k24t2"]
Sep 17 18:49:12.918542 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.918520    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-k24t2"
Sep 17 18:49:12.923200 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.923180    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:49:12.923562 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.923544    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4wtpq\""
Sep 17 18:49:12.931788 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:12.931764    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-k24t2"]
Sep 17 18:49:13.001161 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.001128    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvpt6\" (UniqueName: \"kubernetes.io/projected/7df9edab-0c07-444d-9c7b-683d6599929a-kube-api-access-fvpt6\") pod \"kserve-controller-manager-7dcf484667-jgqhq\" (UID: \"7df9edab-0c07-444d-9c7b-683d6599929a\") " pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:13.001357 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.001175    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtr7j\" (UniqueName: \"kubernetes.io/projected/dda9b6ff-71bf-4dcc-9050-513c47e80e8a-kube-api-access-wtr7j\") pod \"seaweedfs-64568bcd49-k24t2\" (UID: \"dda9b6ff-71bf-4dcc-9050-513c47e80e8a\") " pod="kserve/seaweedfs-64568bcd49-k24t2"
Sep 17 18:49:13.001357 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.001210    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7df9edab-0c07-444d-9c7b-683d6599929a-cert\") pod \"kserve-controller-manager-7dcf484667-jgqhq\" (UID: \"7df9edab-0c07-444d-9c7b-683d6599929a\") " pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:13.001357 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.001235    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dda9b6ff-71bf-4dcc-9050-513c47e80e8a-data\") pod \"seaweedfs-64568bcd49-k24t2\" (UID: \"dda9b6ff-71bf-4dcc-9050-513c47e80e8a\") " pod="kserve/seaweedfs-64568bcd49-k24t2"
Sep 17 18:49:13.001357 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:49:13.001336    2586 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 18:49:13.001557 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:49:13.001395    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df9edab-0c07-444d-9c7b-683d6599929a-cert podName:7df9edab-0c07-444d-9c7b-683d6599929a nodeName:}" failed. No retries permitted until 2026-09-17 18:49:13.501379176 +0000 UTC m=+574.361203831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7df9edab-0c07-444d-9c7b-683d6599929a-cert") pod "kserve-controller-manager-7dcf484667-jgqhq" (UID: "7df9edab-0c07-444d-9c7b-683d6599929a") : secret "kserve-webhook-server-cert" not found
Sep 17 18:49:13.010597 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.010574    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvpt6\" (UniqueName: \"kubernetes.io/projected/7df9edab-0c07-444d-9c7b-683d6599929a-kube-api-access-fvpt6\") pod \"kserve-controller-manager-7dcf484667-jgqhq\" (UID: \"7df9edab-0c07-444d-9c7b-683d6599929a\") " pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:13.102204 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.102109    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dda9b6ff-71bf-4dcc-9050-513c47e80e8a-data\") pod \"seaweedfs-64568bcd49-k24t2\" (UID: \"dda9b6ff-71bf-4dcc-9050-513c47e80e8a\") " pod="kserve/seaweedfs-64568bcd49-k24t2"
Sep 17 18:49:13.102204 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.102183    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtr7j\" (UniqueName: \"kubernetes.io/projected/dda9b6ff-71bf-4dcc-9050-513c47e80e8a-kube-api-access-wtr7j\") pod \"seaweedfs-64568bcd49-k24t2\" (UID: \"dda9b6ff-71bf-4dcc-9050-513c47e80e8a\") " pod="kserve/seaweedfs-64568bcd49-k24t2"
Sep 17 18:49:13.102504 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.102486    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dda9b6ff-71bf-4dcc-9050-513c47e80e8a-data\") pod \"seaweedfs-64568bcd49-k24t2\" (UID: \"dda9b6ff-71bf-4dcc-9050-513c47e80e8a\") " pod="kserve/seaweedfs-64568bcd49-k24t2"
Sep 17 18:49:13.119743 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.119721    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtr7j\" (UniqueName: \"kubernetes.io/projected/dda9b6ff-71bf-4dcc-9050-513c47e80e8a-kube-api-access-wtr7j\") pod \"seaweedfs-64568bcd49-k24t2\" (UID: \"dda9b6ff-71bf-4dcc-9050-513c47e80e8a\") " pod="kserve/seaweedfs-64568bcd49-k24t2"
Sep 17 18:49:13.227649 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.227615    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-k24t2"
Sep 17 18:49:13.505238 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.505146    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7df9edab-0c07-444d-9c7b-683d6599929a-cert\") pod \"kserve-controller-manager-7dcf484667-jgqhq\" (UID: \"7df9edab-0c07-444d-9c7b-683d6599929a\") " pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:13.507646 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.507622    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7df9edab-0c07-444d-9c7b-683d6599929a-cert\") pod \"kserve-controller-manager-7dcf484667-jgqhq\" (UID: \"7df9edab-0c07-444d-9c7b-683d6599929a\") " pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:13.562057 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.562033    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-k24t2"]
Sep 17 18:49:13.562920 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:49:13.562898    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda9b6ff_71bf_4dcc_9050_513c47e80e8a.slice/crio-bf740b8a2cb962e427627d8c07584805b0a1afa8ca479df10793a001662021b2 WatchSource:0}: Error finding container bf740b8a2cb962e427627d8c07584805b0a1afa8ca479df10793a001662021b2: Status 404 returned error can't find the container with id bf740b8a2cb962e427627d8c07584805b0a1afa8ca479df10793a001662021b2
Sep 17 18:49:13.769294 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:13.769206    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:14.042529 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:14.042337    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf484667-jgqhq"]
Sep 17 18:49:14.044975 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:49:14.044948    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df9edab_0c07_444d_9c7b_683d6599929a.slice/crio-8f505d2f70ef641383b3052b330463fc266ff3e83e7f517b1dd15e1075a99993 WatchSource:0}: Error finding container 8f505d2f70ef641383b3052b330463fc266ff3e83e7f517b1dd15e1075a99993: Status 404 returned error can't find the container with id 8f505d2f70ef641383b3052b330463fc266ff3e83e7f517b1dd15e1075a99993
Sep 17 18:49:14.113573 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:14.113526    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq" event={"ID":"7df9edab-0c07-444d-9c7b-683d6599929a","Type":"ContainerStarted","Data":"8f505d2f70ef641383b3052b330463fc266ff3e83e7f517b1dd15e1075a99993"}
Sep 17 18:49:14.115209 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:14.115172    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-k24t2" event={"ID":"dda9b6ff-71bf-4dcc-9050-513c47e80e8a","Type":"ContainerStarted","Data":"bf740b8a2cb962e427627d8c07584805b0a1afa8ca479df10793a001662021b2"}
Sep 17 18:49:18.146604 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:18.146563    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-k24t2" event={"ID":"dda9b6ff-71bf-4dcc-9050-513c47e80e8a","Type":"ContainerStarted","Data":"52ec0b9e3f296ed1a1ff40f49530d23162e766344669f86611b099b30d0ab73d"}
Sep 17 18:49:18.147093 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:18.146761    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-k24t2"
Sep 17 18:49:18.148212 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:18.148186    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq" event={"ID":"7df9edab-0c07-444d-9c7b-683d6599929a","Type":"ContainerStarted","Data":"3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a"}
Sep 17 18:49:18.148345 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:18.148248    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:18.166362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:18.166313    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-k24t2" podStartSLOduration=2.128238055 podStartE2EDuration="6.166296651s" totalImagesPullingTime="4.038058596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:49:18.163981965 +0000 UTC m=+579.023806628" watchObservedRunningTime="2026-09-17 18:49:18.166296651 +0000 UTC m=+579.026121344"
Sep 17 18:49:18.183482 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:18.183432    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq" podStartSLOduration=2.719649721 podStartE2EDuration="6.183415497s" totalImagesPullingTime="3.463765776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:49:18.181784786 +0000 UTC m=+579.041609457" watchObservedRunningTime="2026-09-17 18:49:18.183415497 +0000 UTC m=+579.043240145"
Sep 17 18:49:24.156806 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:24.156773    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-k24t2"
Sep 17 18:49:41.350542 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:41.350511    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:49:41.351498 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:41.351477    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:49:49.159684 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:49.159653    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:56.430145 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.430050    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7dcf484667-jgqhq"]
Sep 17 18:49:56.430616 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.430321    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq" podUID="7df9edab-0c07-444d-9c7b-683d6599929a" containerName="manager" containerID="cri-o://3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a" gracePeriod=10
Sep 17 18:49:56.456445 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.456414    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7dcf484667-bf8ww"]
Sep 17 18:49:56.459146 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.459126    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf484667-bf8ww"
Sep 17 18:49:56.468640 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.468597    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf484667-bf8ww"]
Sep 17 18:49:56.583077 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.583034    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvbc\" (UniqueName: \"kubernetes.io/projected/2acd3526-1681-4bee-b51f-7f51a7562721-kube-api-access-dsvbc\") pod \"kserve-controller-manager-7dcf484667-bf8ww\" (UID: \"2acd3526-1681-4bee-b51f-7f51a7562721\") " pod="kserve/kserve-controller-manager-7dcf484667-bf8ww"
Sep 17 18:49:56.583280 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.583182    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2acd3526-1681-4bee-b51f-7f51a7562721-cert\") pod \"kserve-controller-manager-7dcf484667-bf8ww\" (UID: \"2acd3526-1681-4bee-b51f-7f51a7562721\") " pod="kserve/kserve-controller-manager-7dcf484667-bf8ww"
Sep 17 18:49:56.668315 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.668289    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:56.684628 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.684546    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvbc\" (UniqueName: \"kubernetes.io/projected/2acd3526-1681-4bee-b51f-7f51a7562721-kube-api-access-dsvbc\") pod \"kserve-controller-manager-7dcf484667-bf8ww\" (UID: \"2acd3526-1681-4bee-b51f-7f51a7562721\") " pod="kserve/kserve-controller-manager-7dcf484667-bf8ww"
Sep 17 18:49:56.684766 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.684668    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2acd3526-1681-4bee-b51f-7f51a7562721-cert\") pod \"kserve-controller-manager-7dcf484667-bf8ww\" (UID: \"2acd3526-1681-4bee-b51f-7f51a7562721\") " pod="kserve/kserve-controller-manager-7dcf484667-bf8ww"
Sep 17 18:49:56.687253 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.687228    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2acd3526-1681-4bee-b51f-7f51a7562721-cert\") pod \"kserve-controller-manager-7dcf484667-bf8ww\" (UID: \"2acd3526-1681-4bee-b51f-7f51a7562721\") " pod="kserve/kserve-controller-manager-7dcf484667-bf8ww"
Sep 17 18:49:56.698369 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.698336    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvbc\" (UniqueName: \"kubernetes.io/projected/2acd3526-1681-4bee-b51f-7f51a7562721-kube-api-access-dsvbc\") pod \"kserve-controller-manager-7dcf484667-bf8ww\" (UID: \"2acd3526-1681-4bee-b51f-7f51a7562721\") " pod="kserve/kserve-controller-manager-7dcf484667-bf8ww"
Sep 17 18:49:56.785304 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.785273    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7df9edab-0c07-444d-9c7b-683d6599929a-cert\") pod \"7df9edab-0c07-444d-9c7b-683d6599929a\" (UID: \"7df9edab-0c07-444d-9c7b-683d6599929a\") "
Sep 17 18:49:56.785517 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.785314    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvpt6\" (UniqueName: \"kubernetes.io/projected/7df9edab-0c07-444d-9c7b-683d6599929a-kube-api-access-fvpt6\") pod \"7df9edab-0c07-444d-9c7b-683d6599929a\" (UID: \"7df9edab-0c07-444d-9c7b-683d6599929a\") "
Sep 17 18:49:56.787662 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.787636    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df9edab-0c07-444d-9c7b-683d6599929a-cert" (OuterVolumeSpecName: "cert") pod "7df9edab-0c07-444d-9c7b-683d6599929a" (UID: "7df9edab-0c07-444d-9c7b-683d6599929a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:56.787769 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.787702    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df9edab-0c07-444d-9c7b-683d6599929a-kube-api-access-fvpt6" (OuterVolumeSpecName: "kube-api-access-fvpt6") pod "7df9edab-0c07-444d-9c7b-683d6599929a" (UID: "7df9edab-0c07-444d-9c7b-683d6599929a"). InnerVolumeSpecName "kube-api-access-fvpt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:56.806043 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.806017    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf484667-bf8ww"
Sep 17 18:49:56.886472 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.886444    2586 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7df9edab-0c07-444d-9c7b-683d6599929a-cert\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:49:56.886472 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:56.886470    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvpt6\" (UniqueName: \"kubernetes.io/projected/7df9edab-0c07-444d-9c7b-683d6599929a-kube-api-access-fvpt6\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:49:57.077906 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.077876    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf484667-bf8ww"]
Sep 17 18:49:57.081435 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:49:57.081408    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acd3526_1681_4bee_b51f_7f51a7562721.slice/crio-02c01bbece462f6050b81a8401b4722109dd21ba1471f76ff139b3f2a78248be WatchSource:0}: Error finding container 02c01bbece462f6050b81a8401b4722109dd21ba1471f76ff139b3f2a78248be: Status 404 returned error can't find the container with id 02c01bbece462f6050b81a8401b4722109dd21ba1471f76ff139b3f2a78248be
Sep 17 18:49:57.082645 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.082629    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:49:57.357328 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.357234    2586 generic.go:356] "Generic (PLEG): container finished" podID="7df9edab-0c07-444d-9c7b-683d6599929a" containerID="3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a" exitCode=0
Sep 17 18:49:57.357328 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.357282    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq"
Sep 17 18:49:57.357573 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.357329    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq" event={"ID":"7df9edab-0c07-444d-9c7b-683d6599929a","Type":"ContainerDied","Data":"3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a"}
Sep 17 18:49:57.357573 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.357392    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf484667-jgqhq" event={"ID":"7df9edab-0c07-444d-9c7b-683d6599929a","Type":"ContainerDied","Data":"8f505d2f70ef641383b3052b330463fc266ff3e83e7f517b1dd15e1075a99993"}
Sep 17 18:49:57.357573 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.357422    2586 scope.go:117] "RemoveContainer" containerID="3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a"
Sep 17 18:49:57.358659 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.358641    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf484667-bf8ww" event={"ID":"2acd3526-1681-4bee-b51f-7f51a7562721","Type":"ContainerStarted","Data":"02c01bbece462f6050b81a8401b4722109dd21ba1471f76ff139b3f2a78248be"}
Sep 17 18:49:57.367240 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.367224    2586 scope.go:117] "RemoveContainer" containerID="3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a"
Sep 17 18:49:57.367519 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:49:57.367500    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a\": container with ID starting with 3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a not found: ID does not exist" containerID="3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a"
Sep 17 18:49:57.367572 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.367535    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a"} err="failed to get container status \"3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a\": rpc error: code = NotFound desc = could not find container \"3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a\": container with ID starting with 3c6635d736193f5e6990012167334af03f9acad72d3e88da722cc0aee97baa0a not found: ID does not exist"
Sep 17 18:49:57.379426 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.379403    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7dcf484667-jgqhq"]
Sep 17 18:49:57.386462 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.386434    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7dcf484667-jgqhq"]
Sep 17 18:49:57.459751 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:57.459720    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df9edab-0c07-444d-9c7b-683d6599929a" path="/var/lib/kubelet/pods/7df9edab-0c07-444d-9c7b-683d6599929a/volumes"
Sep 17 18:49:58.366283 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:58.366247    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf484667-bf8ww" event={"ID":"2acd3526-1681-4bee-b51f-7f51a7562721","Type":"ContainerStarted","Data":"0644eebcf7b78e0609438f77ddac1794f90b45a5581a4800ed0869212523a566"}
Sep 17 18:49:58.366481 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:58.366407    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7dcf484667-bf8ww"
Sep 17 18:49:58.386340 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:49:58.386288    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7dcf484667-bf8ww" podStartSLOduration=2.064789737 podStartE2EDuration="2.386267316s" totalImagesPullingTime="321.477579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:49:58.384370857 +0000 UTC m=+619.244195517" watchObservedRunningTime="2026-09-17 18:49:58.386267316 +0000 UTC m=+619.246091966"
Sep 17 18:50:29.377039 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:29.377006    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7dcf484667-bf8ww"
Sep 17 18:50:30.798251 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.797443    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-wnh9c"]
Sep 17 18:50:30.798251 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.798006    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df9edab-0c07-444d-9c7b-683d6599929a" containerName="manager"
Sep 17 18:50:30.798251 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.798024    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df9edab-0c07-444d-9c7b-683d6599929a" containerName="manager"
Sep 17 18:50:30.798251 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.798107    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df9edab-0c07-444d-9c7b-683d6599929a" containerName="manager"
Sep 17 18:50:30.801903 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.801876    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:50:30.804637 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.804615    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 18:50:30.805118 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.805103    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-tdbr5\""
Sep 17 18:50:30.816991 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.816961    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-wnh9c"]
Sep 17 18:50:30.822705 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.822683    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-2n5nn"]
Sep 17 18:50:30.826277 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.826259    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:30.829807 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.829700    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-sv5qq\""
Sep 17 18:50:30.831389 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.831365    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 18:50:30.831515 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.831366    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 18:50:30.847307 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.847271    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-2n5nn"]
Sep 17 18:50:30.899935 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.899898    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48-tls-certs\") pod \"model-serving-api-7b954ffc95-wnh9c\" (UID: \"34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48\") " pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:50:30.900121 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:30.899969    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qg88\" (UniqueName: \"kubernetes.io/projected/34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48-kube-api-access-9qg88\") pod \"model-serving-api-7b954ffc95-wnh9c\" (UID: \"34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48\") " pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:50:31.001441 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.001399    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shw9m\" (UniqueName: \"kubernetes.io/projected/4bb34e51-7e61-4c03-b670-1c34b495372c-kube-api-access-shw9m\") pod \"odh-model-controller-7b966d7984-2n5nn\" (UID: \"4bb34e51-7e61-4c03-b670-1c34b495372c\") " pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:31.001652 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.001457    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qg88\" (UniqueName: \"kubernetes.io/projected/34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48-kube-api-access-9qg88\") pod \"model-serving-api-7b954ffc95-wnh9c\" (UID: \"34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48\") " pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:50:31.001652 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.001528    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb34e51-7e61-4c03-b670-1c34b495372c-metrics-certs\") pod \"odh-model-controller-7b966d7984-2n5nn\" (UID: \"4bb34e51-7e61-4c03-b670-1c34b495372c\") " pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:31.001652 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.001576    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb34e51-7e61-4c03-b670-1c34b495372c-cert\") pod \"odh-model-controller-7b966d7984-2n5nn\" (UID: \"4bb34e51-7e61-4c03-b670-1c34b495372c\") " pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:31.001652 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.001610    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48-tls-certs\") pod \"model-serving-api-7b954ffc95-wnh9c\" (UID: \"34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48\") " pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:50:31.001883 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:50:31.001722    2586 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 18:50:31.001883 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:50:31.001785    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48-tls-certs podName:34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48 nodeName:}" failed. No retries permitted until 2026-09-17 18:50:31.501767237 +0000 UTC m=+652.361591862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48-tls-certs") pod "model-serving-api-7b954ffc95-wnh9c" (UID: "34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48") : secret "model-serving-api-tls" not found
Sep 17 18:50:31.011469 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.011438    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qg88\" (UniqueName: \"kubernetes.io/projected/34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48-kube-api-access-9qg88\") pod \"model-serving-api-7b954ffc95-wnh9c\" (UID: \"34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48\") " pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:50:31.102672 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.102638    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb34e51-7e61-4c03-b670-1c34b495372c-cert\") pod \"odh-model-controller-7b966d7984-2n5nn\" (UID: \"4bb34e51-7e61-4c03-b670-1c34b495372c\") " pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:31.102922 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.102721    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shw9m\" (UniqueName: \"kubernetes.io/projected/4bb34e51-7e61-4c03-b670-1c34b495372c-kube-api-access-shw9m\") pod \"odh-model-controller-7b966d7984-2n5nn\" (UID: \"4bb34e51-7e61-4c03-b670-1c34b495372c\") " pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:31.102922 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.102762    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb34e51-7e61-4c03-b670-1c34b495372c-metrics-certs\") pod \"odh-model-controller-7b966d7984-2n5nn\" (UID: \"4bb34e51-7e61-4c03-b670-1c34b495372c\") " pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:31.105420 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.105395    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb34e51-7e61-4c03-b670-1c34b495372c-metrics-certs\") pod \"odh-model-controller-7b966d7984-2n5nn\" (UID: \"4bb34e51-7e61-4c03-b670-1c34b495372c\") " pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:31.105524 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.105445    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb34e51-7e61-4c03-b670-1c34b495372c-cert\") pod \"odh-model-controller-7b966d7984-2n5nn\" (UID: \"4bb34e51-7e61-4c03-b670-1c34b495372c\") " pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:31.112400 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.112377    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shw9m\" (UniqueName: \"kubernetes.io/projected/4bb34e51-7e61-4c03-b670-1c34b495372c-kube-api-access-shw9m\") pod \"odh-model-controller-7b966d7984-2n5nn\" (UID: \"4bb34e51-7e61-4c03-b670-1c34b495372c\") " pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:31.138007 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.137974    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:31.419876 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.419850    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-2n5nn"]
Sep 17 18:50:31.422240 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:50:31.422214    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb34e51_7e61_4c03_b670_1c34b495372c.slice/crio-7f32c753fc4049f70b2890729ed2c83abcf44788b523c186e645d379861ae50b WatchSource:0}: Error finding container 7f32c753fc4049f70b2890729ed2c83abcf44788b523c186e645d379861ae50b: Status 404 returned error can't find the container with id 7f32c753fc4049f70b2890729ed2c83abcf44788b523c186e645d379861ae50b
Sep 17 18:50:31.506218 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.506175    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48-tls-certs\") pod \"model-serving-api-7b954ffc95-wnh9c\" (UID: \"34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48\") " pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:50:31.508806 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.508783    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48-tls-certs\") pod \"model-serving-api-7b954ffc95-wnh9c\" (UID: \"34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48\") " pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:50:31.539666 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.539631    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-2n5nn" event={"ID":"4bb34e51-7e61-4c03-b670-1c34b495372c","Type":"ContainerStarted","Data":"7f32c753fc4049f70b2890729ed2c83abcf44788b523c186e645d379861ae50b"}
Sep 17 18:50:31.711908 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.711794    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:50:31.991805 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:31.991771    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-wnh9c"]
Sep 17 18:50:31.993380 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:50:31.993338    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a7e137_a8dd_4f9e_b417_7cf6a0cfaa48.slice/crio-49f2c565fc60746f73cbb2daab4cae2361651067cbfeb05ac21b2a0c79166b6e WatchSource:0}: Error finding container 49f2c565fc60746f73cbb2daab4cae2361651067cbfeb05ac21b2a0c79166b6e: Status 404 returned error can't find the container with id 49f2c565fc60746f73cbb2daab4cae2361651067cbfeb05ac21b2a0c79166b6e
Sep 17 18:50:32.549724 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:32.549648    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-wnh9c" event={"ID":"34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48","Type":"ContainerStarted","Data":"49f2c565fc60746f73cbb2daab4cae2361651067cbfeb05ac21b2a0c79166b6e"}
Sep 17 18:50:35.569321 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:35.569280    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-2n5nn" event={"ID":"4bb34e51-7e61-4c03-b670-1c34b495372c","Type":"ContainerStarted","Data":"b6b49c50de9ee98bbaaea4a6e42e2186b87df72fe478e409d6098cb4e6582862"}
Sep 17 18:50:35.569770 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:35.569406    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:35.571079 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:35.571047    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-wnh9c" event={"ID":"34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48","Type":"ContainerStarted","Data":"98e1a158c801482283d712c8496eb3f1b0292aea32be8997ebde2e15b167decc"}
Sep 17 18:50:35.571219 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:35.571110    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:50:35.590691 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:35.590607    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-2n5nn" podStartSLOduration=1.909848403 podStartE2EDuration="5.590592086s" totalImagesPullingTime="3.680743683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:35.588054097 +0000 UTC m=+656.447878768" watchObservedRunningTime="2026-09-17 18:50:35.590592086 +0000 UTC m=+656.450416734"
Sep 17 18:50:35.608810 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:35.608760    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-wnh9c" podStartSLOduration=2.458359493 podStartE2EDuration="5.608741819s" totalImagesPullingTime="3.150382326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:35.607270138 +0000 UTC m=+656.467094786" watchObservedRunningTime="2026-09-17 18:50:35.608741819 +0000 UTC m=+656.468566467"
Sep 17 18:50:46.579303 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:46.579269    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-2n5nn"
Sep 17 18:50:46.580780 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:50:46.580755    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-wnh9c"
Sep 17 18:52:27.916755 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:27.916720    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"]
Sep 17 18:52:27.920089 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:27.920069    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:27.923015 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:27.922990    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-5rcdt\""
Sep 17 18:52:27.923132 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:27.922990    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 18:52:27.924020 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:27.924001    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 18:52:27.924141 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:27.924058    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 18:52:27.932896 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:27.932872    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"]
Sep 17 18:52:28.032120 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.032072    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f66ebf-f898-4758-9bbb-9efea8093324-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.032120 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.032111    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9cw\" (UniqueName: \"kubernetes.io/projected/53f66ebf-f898-4758-9bbb-9efea8093324-kube-api-access-2f9cw\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.032345 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.032132    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.032345 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.032197    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.032345 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.032257    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.032345 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.032276    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.132763 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.132723    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.132763 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.132760    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.133059 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.132825    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f66ebf-f898-4758-9bbb-9efea8093324-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.133059 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.132856    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9cw\" (UniqueName: \"kubernetes.io/projected/53f66ebf-f898-4758-9bbb-9efea8093324-kube-api-access-2f9cw\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.133059 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.132875    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.133059 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.133008    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.133284 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.133212    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.133284 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.133267    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.133362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.133337    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.135348 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.135323    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.135659 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.135639    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f66ebf-f898-4758-9bbb-9efea8093324-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.141231 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.141209    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9cw\" (UniqueName: \"kubernetes.io/projected/53f66ebf-f898-4758-9bbb-9efea8093324-kube-api-access-2f9cw\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.232180 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.232083    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:28.509515 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:28.509488    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"]
Sep 17 18:52:28.511495 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:52:28.511468    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f66ebf_f898_4758_9bbb_9efea8093324.slice/crio-ad043fdc1d8d57e1e672a2b4fef5e550192fa6edac660dd610d915b00920f914 WatchSource:0}: Error finding container ad043fdc1d8d57e1e672a2b4fef5e550192fa6edac660dd610d915b00920f914: Status 404 returned error can't find the container with id ad043fdc1d8d57e1e672a2b4fef5e550192fa6edac660dd610d915b00920f914
Sep 17 18:52:29.195978 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:29.195948    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8" event={"ID":"53f66ebf-f898-4758-9bbb-9efea8093324","Type":"ContainerStarted","Data":"ad043fdc1d8d57e1e672a2b4fef5e550192fa6edac660dd610d915b00920f914"}
Sep 17 18:52:30.203914 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:30.203878    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8" event={"ID":"53f66ebf-f898-4758-9bbb-9efea8093324","Type":"ContainerStarted","Data":"c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6"}
Sep 17 18:52:30.223732 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:30.223679    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8" podStartSLOduration=1.619756556 podStartE2EDuration="3.22366068s" totalImagesPullingTime="1.603904124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:30.222631867 +0000 UTC m=+771.082456532" watchObservedRunningTime="2026-09-17 18:52:30.22366068 +0000 UTC m=+771.083485327"
Sep 17 18:52:30.233123 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:30.233094    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:37.223023 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.222988    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"]
Sep 17 18:52:37.227879 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.227853    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:52:37.231559 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.231533    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-gtw6r\""
Sep 17 18:52:37.231991 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.231974    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 18:52:37.232922 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.232898    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-pgjjs\""
Sep 17 18:52:37.233048 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.232925    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 18:52:37.233048 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.232901    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 18:52:37.236585 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.236560    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"]
Sep 17 18:52:37.315124 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.315087    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde6220e-4a37-4819-a753-fe6db24e0330-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2\" (UID: \"bde6220e-4a37-4819-a753-fe6db24e0330\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:52:37.315278 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.315133    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grk77\" (UniqueName: \"kubernetes.io/projected/bde6220e-4a37-4819-a753-fe6db24e0330-kube-api-access-grk77\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2\" (UID: \"bde6220e-4a37-4819-a753-fe6db24e0330\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:52:37.415602 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.415564    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde6220e-4a37-4819-a753-fe6db24e0330-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2\" (UID: \"bde6220e-4a37-4819-a753-fe6db24e0330\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:52:37.415784 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.415620    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grk77\" (UniqueName: \"kubernetes.io/projected/bde6220e-4a37-4819-a753-fe6db24e0330-kube-api-access-grk77\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2\" (UID: \"bde6220e-4a37-4819-a753-fe6db24e0330\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:52:37.418319 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.418289    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde6220e-4a37-4819-a753-fe6db24e0330-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2\" (UID: \"bde6220e-4a37-4819-a753-fe6db24e0330\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:52:37.424151 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.424124    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grk77\" (UniqueName: \"kubernetes.io/projected/bde6220e-4a37-4819-a753-fe6db24e0330-kube-api-access-grk77\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2\" (UID: \"bde6220e-4a37-4819-a753-fe6db24e0330\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:52:37.541781 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.541674    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:52:37.819701 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:37.819667    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"]
Sep 17 18:52:37.822595 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:52:37.822565    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde6220e_4a37_4819_a753_fe6db24e0330.slice/crio-0bf3baaffcdc82642f69334ed2413ffb160bf68134050fe1d010e0c0aa0f7789 WatchSource:0}: Error finding container 0bf3baaffcdc82642f69334ed2413ffb160bf68134050fe1d010e0c0aa0f7789: Status 404 returned error can't find the container with id 0bf3baaffcdc82642f69334ed2413ffb160bf68134050fe1d010e0c0aa0f7789
Sep 17 18:52:38.232719 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:38.232627    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:38.245799 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:38.245776    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:38.251603 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:38.251579    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" event={"ID":"bde6220e-4a37-4819-a753-fe6db24e0330","Type":"ContainerStarted","Data":"0bf3baaffcdc82642f69334ed2413ffb160bf68134050fe1d010e0c0aa0f7789"}
Sep 17 18:52:38.263200 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:38.263176    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:52:40.267177 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:40.267144    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" event={"ID":"bde6220e-4a37-4819-a753-fe6db24e0330","Type":"ContainerStarted","Data":"517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae"}
Sep 17 18:52:40.267528 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:40.267358    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:52:40.289412 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:52:40.289360    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" podStartSLOduration=0.927385389 podStartE2EDuration="3.289342552s" totalImagesPullingTime="2.361957163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:40.28706521 +0000 UTC m=+781.146889879" watchObservedRunningTime="2026-09-17 18:52:40.289342552 +0000 UTC m=+781.149167199"
Sep 17 18:53:10.809316 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:10.809281    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"]
Sep 17 18:53:10.819108 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:53:10.819069    2586 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:53:10.819108 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:53:10.819108 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 18:53:10.819108 ip-10-0-129-204 kubenswrapper[2586]:  > containerID="c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6" cmd=["/bin/sleep","15"]
Sep 17 18:53:10.819620 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:10.819137    2586 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8" message="" err=<
Sep 17 18:53:10.819620 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:53:10.819620 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 18:53:10.819620 ip-10-0-129-204 kubenswrapper[2586]:  >
Sep 17 18:53:10.819620 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:53:10.819153    2586 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:53:10.819620 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:53:10.819620 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 18:53:10.819620 ip-10-0-129-204 kubenswrapper[2586]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8" podUID="53f66ebf-f898-4758-9bbb-9efea8093324" containerName="main" containerID="cri-o://c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6"
Sep 17 18:53:10.819620 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:10.819198    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8" podUID="53f66ebf-f898-4758-9bbb-9efea8093324" containerName="main" containerID="cri-o://c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6" gracePeriod=60
Sep 17 18:53:11.175295 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.175268    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:53:11.223022 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.222984    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-dshm\") pod \"53f66ebf-f898-4758-9bbb-9efea8093324\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") "
Sep 17 18:53:11.223228 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.223054    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f66ebf-f898-4758-9bbb-9efea8093324-tls-certs\") pod \"53f66ebf-f898-4758-9bbb-9efea8093324\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") "
Sep 17 18:53:11.223228 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.223080    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f9cw\" (UniqueName: \"kubernetes.io/projected/53f66ebf-f898-4758-9bbb-9efea8093324-kube-api-access-2f9cw\") pod \"53f66ebf-f898-4758-9bbb-9efea8093324\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") "
Sep 17 18:53:11.223228 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.223221    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-home\") pod \"53f66ebf-f898-4758-9bbb-9efea8093324\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") "
Sep 17 18:53:11.223391 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.223299    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-model-cache\") pod \"53f66ebf-f898-4758-9bbb-9efea8093324\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") "
Sep 17 18:53:11.223391 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.223319    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-tmp-dir\") pod \"53f66ebf-f898-4758-9bbb-9efea8093324\" (UID: \"53f66ebf-f898-4758-9bbb-9efea8093324\") "
Sep 17 18:53:11.223558 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.223536    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-home" (OuterVolumeSpecName: "home") pod "53f66ebf-f898-4758-9bbb-9efea8093324" (UID: "53f66ebf-f898-4758-9bbb-9efea8093324"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:11.223633 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.223567    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-model-cache" (OuterVolumeSpecName: "model-cache") pod "53f66ebf-f898-4758-9bbb-9efea8093324" (UID: "53f66ebf-f898-4758-9bbb-9efea8093324"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:11.223700 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.223673    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53f66ebf-f898-4758-9bbb-9efea8093324" (UID: "53f66ebf-f898-4758-9bbb-9efea8093324"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:11.225586 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.225557    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-dshm" (OuterVolumeSpecName: "dshm") pod "53f66ebf-f898-4758-9bbb-9efea8093324" (UID: "53f66ebf-f898-4758-9bbb-9efea8093324"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:11.225586 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.225561    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f66ebf-f898-4758-9bbb-9efea8093324-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53f66ebf-f898-4758-9bbb-9efea8093324" (UID: "53f66ebf-f898-4758-9bbb-9efea8093324"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:11.225741 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.225661    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f66ebf-f898-4758-9bbb-9efea8093324-kube-api-access-2f9cw" (OuterVolumeSpecName: "kube-api-access-2f9cw") pod "53f66ebf-f898-4758-9bbb-9efea8093324" (UID: "53f66ebf-f898-4758-9bbb-9efea8093324"). InnerVolumeSpecName "kube-api-access-2f9cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:11.274499 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.274468    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:53:11.324205 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.324160    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:53:11.324205 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.324188    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:53:11.324205 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.324201    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:53:11.324205 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.324213    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f66ebf-f898-4758-9bbb-9efea8093324-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:53:11.324626 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.324224    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9cw\" (UniqueName: \"kubernetes.io/projected/53f66ebf-f898-4758-9bbb-9efea8093324-kube-api-access-2f9cw\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:53:11.324626 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.324236    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53f66ebf-f898-4758-9bbb-9efea8093324-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:53:11.444726 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.444634    2586 generic.go:356] "Generic (PLEG): container finished" podID="53f66ebf-f898-4758-9bbb-9efea8093324" containerID="c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6" exitCode=0
Sep 17 18:53:11.444888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.444728    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"
Sep 17 18:53:11.444888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.444766    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8" event={"ID":"53f66ebf-f898-4758-9bbb-9efea8093324","Type":"ContainerDied","Data":"c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6"}
Sep 17 18:53:11.444888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.444826    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8" event={"ID":"53f66ebf-f898-4758-9bbb-9efea8093324","Type":"ContainerDied","Data":"ad043fdc1d8d57e1e672a2b4fef5e550192fa6edac660dd610d915b00920f914"}
Sep 17 18:53:11.444888 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.444866    2586 scope.go:117] "RemoveContainer" containerID="c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6"
Sep 17 18:53:11.455695 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.455675    2586 scope.go:117] "RemoveContainer" containerID="c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6"
Sep 17 18:53:11.456071 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:53:11.456050    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6\": container with ID starting with c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6 not found: ID does not exist" containerID="c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6"
Sep 17 18:53:11.456161 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.456079    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6"} err="failed to get container status \"c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6\": rpc error: code = NotFound desc = could not find container \"c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6\": container with ID starting with c3ddbf3281dd95fe47a24546a6eded4009174392c4e7c2bf6b1febd73f1dd6f6 not found: ID does not exist"
Sep 17 18:53:11.474485 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.474453    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"]
Sep 17 18:53:11.478666 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:11.478644    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9pkd8"]
Sep 17 18:53:13.465204 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:13.465162    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53f66ebf-f898-4758-9bbb-9efea8093324" path="/var/lib/kubelet/pods/53f66ebf-f898-4758-9bbb-9efea8093324/volumes"
Sep 17 18:53:18.056076 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:18.056045    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"]
Sep 17 18:53:21.274447 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:21.274410    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" podUID="bde6220e-4a37-4819-a753-fe6db24e0330" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:31.274988 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:31.274945    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" podUID="bde6220e-4a37-4819-a753-fe6db24e0330" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:33.076404 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:33.076364    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" podUID="bde6220e-4a37-4819-a753-fe6db24e0330" containerName="main" containerID="cri-o://517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae" gracePeriod=45
Sep 17 18:53:41.274713 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:41.274675    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" podUID="bde6220e-4a37-4819-a753-fe6db24e0330" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:41.275206 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:41.274797    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:53:51.274190 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:53:51.274150    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" podUID="bde6220e-4a37-4819-a753-fe6db24e0330" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:01.274751 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:01.274713    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" podUID="bde6220e-4a37-4819-a753-fe6db24e0330" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:03.124023 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:54:03.123990    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde6220e_4a37_4819_a753_fe6db24e0330.slice/crio-conmon-517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 18:54:03.322704 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.322681    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:54:03.376156 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.376070    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grk77\" (UniqueName: \"kubernetes.io/projected/bde6220e-4a37-4819-a753-fe6db24e0330-kube-api-access-grk77\") pod \"bde6220e-4a37-4819-a753-fe6db24e0330\" (UID: \"bde6220e-4a37-4819-a753-fe6db24e0330\") "
Sep 17 18:54:03.376156 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.376111    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde6220e-4a37-4819-a753-fe6db24e0330-tls-certs\") pod \"bde6220e-4a37-4819-a753-fe6db24e0330\" (UID: \"bde6220e-4a37-4819-a753-fe6db24e0330\") "
Sep 17 18:54:03.378467 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.378433    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde6220e-4a37-4819-a753-fe6db24e0330-kube-api-access-grk77" (OuterVolumeSpecName: "kube-api-access-grk77") pod "bde6220e-4a37-4819-a753-fe6db24e0330" (UID: "bde6220e-4a37-4819-a753-fe6db24e0330"). InnerVolumeSpecName "kube-api-access-grk77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:03.378585 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.378462    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde6220e-4a37-4819-a753-fe6db24e0330-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bde6220e-4a37-4819-a753-fe6db24e0330" (UID: "bde6220e-4a37-4819-a753-fe6db24e0330"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:03.477394 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.477367    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grk77\" (UniqueName: \"kubernetes.io/projected/bde6220e-4a37-4819-a753-fe6db24e0330-kube-api-access-grk77\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:54:03.477394 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.477394    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde6220e-4a37-4819-a753-fe6db24e0330-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:54:03.736947 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.736828    2586 generic.go:356] "Generic (PLEG): container finished" podID="bde6220e-4a37-4819-a753-fe6db24e0330" containerID="517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae" exitCode=0
Sep 17 18:54:03.736947 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.736915    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"
Sep 17 18:54:03.737197 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.736943    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" event={"ID":"bde6220e-4a37-4819-a753-fe6db24e0330","Type":"ContainerDied","Data":"517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae"}
Sep 17 18:54:03.737197 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.737002    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2" event={"ID":"bde6220e-4a37-4819-a753-fe6db24e0330","Type":"ContainerDied","Data":"0bf3baaffcdc82642f69334ed2413ffb160bf68134050fe1d010e0c0aa0f7789"}
Sep 17 18:54:03.737197 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.737030    2586 scope.go:117] "RemoveContainer" containerID="517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae"
Sep 17 18:54:03.745909 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.745888    2586 scope.go:117] "RemoveContainer" containerID="517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae"
Sep 17 18:54:03.746212 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:54:03.746194    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae\": container with ID starting with 517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae not found: ID does not exist" containerID="517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae"
Sep 17 18:54:03.746262 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.746220    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae"} err="failed to get container status \"517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae\": rpc error: code = NotFound desc = could not find container \"517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae\": container with ID starting with 517112c43bf7ede847db4d2b8a8e59147c54eb1a6616f64eb93dcec432c333ae not found: ID does not exist"
Sep 17 18:54:03.754913 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.754890    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"]
Sep 17 18:54:03.758381 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:03.758360    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-4bhm2"]
Sep 17 18:54:05.458986 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:05.458942    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bde6220e-4a37-4819-a753-fe6db24e0330" path="/var/lib/kubelet/pods/bde6220e-4a37-4819-a753-fe6db24e0330/volumes"
Sep 17 18:54:39.253143 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.253067    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"]
Sep 17 18:54:39.253614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.253443    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f66ebf-f898-4758-9bbb-9efea8093324" containerName="main"
Sep 17 18:54:39.253614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.253458    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f66ebf-f898-4758-9bbb-9efea8093324" containerName="main"
Sep 17 18:54:39.253614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.253487    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bde6220e-4a37-4819-a753-fe6db24e0330" containerName="main"
Sep 17 18:54:39.253614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.253493    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bde6220e-4a37-4819-a753-fe6db24e0330" containerName="main"
Sep 17 18:54:39.253614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.253557    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="bde6220e-4a37-4819-a753-fe6db24e0330" containerName="main"
Sep 17 18:54:39.253614 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.253568    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f66ebf-f898-4758-9bbb-9efea8093324" containerName="main"
Sep 17 18:54:39.256479 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.256459    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:54:39.259785 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.259762    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 18:54:39.259913 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.259763    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 18:54:39.261492 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.261473    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-ksnkm\""
Sep 17 18:54:39.261615 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.261475    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 18:54:39.282613 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.282582    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"]
Sep 17 18:54:39.284511 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.284487    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-zvl2n\""
Sep 17 18:54:39.403259 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.403219    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vj5p\" (UniqueName: \"kubernetes.io/projected/5e387c32-4684-4ea7-8e1f-5a07654ad261-kube-api-access-6vj5p\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk\" (UID: \"5e387c32-4684-4ea7-8e1f-5a07654ad261\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:54:39.403447 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.403332    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e387c32-4684-4ea7-8e1f-5a07654ad261-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk\" (UID: \"5e387c32-4684-4ea7-8e1f-5a07654ad261\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:54:39.504469 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.504376    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e387c32-4684-4ea7-8e1f-5a07654ad261-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk\" (UID: \"5e387c32-4684-4ea7-8e1f-5a07654ad261\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:54:39.504469 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.504452    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vj5p\" (UniqueName: \"kubernetes.io/projected/5e387c32-4684-4ea7-8e1f-5a07654ad261-kube-api-access-6vj5p\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk\" (UID: \"5e387c32-4684-4ea7-8e1f-5a07654ad261\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:54:39.507064 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.507042    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e387c32-4684-4ea7-8e1f-5a07654ad261-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk\" (UID: \"5e387c32-4684-4ea7-8e1f-5a07654ad261\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:54:39.516957 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.516927    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vj5p\" (UniqueName: \"kubernetes.io/projected/5e387c32-4684-4ea7-8e1f-5a07654ad261-kube-api-access-6vj5p\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk\" (UID: \"5e387c32-4684-4ea7-8e1f-5a07654ad261\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:54:39.591697 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.591656    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:54:39.890518 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.890484    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"]
Sep 17 18:54:39.894435 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:54:39.894402    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e387c32_4684_4ea7_8e1f_5a07654ad261.slice/crio-d400e9b8b6db8edd7c3b9298d1c15eb1f55ae4b2165a9b1debca4f6562dcf81b WatchSource:0}: Error finding container d400e9b8b6db8edd7c3b9298d1c15eb1f55ae4b2165a9b1debca4f6562dcf81b: Status 404 returned error can't find the container with id d400e9b8b6db8edd7c3b9298d1c15eb1f55ae4b2165a9b1debca4f6562dcf81b
Sep 17 18:54:39.933214 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:39.933176    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" event={"ID":"5e387c32-4684-4ea7-8e1f-5a07654ad261","Type":"ContainerStarted","Data":"d400e9b8b6db8edd7c3b9298d1c15eb1f55ae4b2165a9b1debca4f6562dcf81b"}
Sep 17 18:54:40.940456 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:40.940420    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" event={"ID":"5e387c32-4684-4ea7-8e1f-5a07654ad261","Type":"ContainerStarted","Data":"03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc"}
Sep 17 18:54:40.940910 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:40.940524    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:54:40.961997 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:40.961942    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" podStartSLOduration=1.961924255 podStartE2EDuration="1.961924255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:40.96090945 +0000 UTC m=+901.820734118" watchObservedRunningTime="2026-09-17 18:54:40.961924255 +0000 UTC m=+901.821748903"
Sep 17 18:54:41.387379 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:41.387352    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:54:41.388592 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:54:41.388575    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:55:11.947884 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:11.947819    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:55:18.334026 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:18.333976    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"]
Sep 17 18:55:21.947773 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:21.947734    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" podUID="5e387c32-4684-4ea7-8e1f-5a07654ad261" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:55:31.948131 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:31.948095    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" podUID="5e387c32-4684-4ea7-8e1f-5a07654ad261" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:55:33.357313 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:33.357247    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" podUID="5e387c32-4684-4ea7-8e1f-5a07654ad261" containerName="main" containerID="cri-o://03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc" gracePeriod=45
Sep 17 18:55:41.947234 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:41.947192    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" podUID="5e387c32-4684-4ea7-8e1f-5a07654ad261" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:55:41.947691 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:41.947296    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:55:49.093989 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.093945    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"]
Sep 17 18:55:49.097636 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.097597    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:55:49.103998 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.103964    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 18:55:49.104153 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.104005    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-ft684\""
Sep 17 18:55:49.104153 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.104059    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-6rxm2\""
Sep 17 18:55:49.104153 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.104060    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 18:55:49.104153 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.103965    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 18:55:49.104362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.104329    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9b2n\" (UniqueName: \"kubernetes.io/projected/b94bb7b1-6c4e-4798-a971-44e9cdef7234-kube-api-access-w9b2n\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h\" (UID: \"b94bb7b1-6c4e-4798-a971-44e9cdef7234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:55:49.104362 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.104355    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b94bb7b1-6c4e-4798-a971-44e9cdef7234-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h\" (UID: \"b94bb7b1-6c4e-4798-a971-44e9cdef7234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:55:49.112802 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.112776    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"]
Sep 17 18:55:49.206111 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.206070    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9b2n\" (UniqueName: \"kubernetes.io/projected/b94bb7b1-6c4e-4798-a971-44e9cdef7234-kube-api-access-w9b2n\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h\" (UID: \"b94bb7b1-6c4e-4798-a971-44e9cdef7234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:55:49.206305 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.206123    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b94bb7b1-6c4e-4798-a971-44e9cdef7234-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h\" (UID: \"b94bb7b1-6c4e-4798-a971-44e9cdef7234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:55:49.208974 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.208946    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b94bb7b1-6c4e-4798-a971-44e9cdef7234-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h\" (UID: \"b94bb7b1-6c4e-4798-a971-44e9cdef7234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:55:49.224758 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.224727    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9b2n\" (UniqueName: \"kubernetes.io/projected/b94bb7b1-6c4e-4798-a971-44e9cdef7234-kube-api-access-w9b2n\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h\" (UID: \"b94bb7b1-6c4e-4798-a971-44e9cdef7234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:55:49.408723 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.408631    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:55:49.688054 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:49.688008    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"]
Sep 17 18:55:49.690662 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:55:49.690636    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94bb7b1_6c4e_4798_a971_44e9cdef7234.slice/crio-dece5f18cd11aceb54139869897bedac4fe8a53b1667e4f357b3304dc773607a WatchSource:0}: Error finding container dece5f18cd11aceb54139869897bedac4fe8a53b1667e4f357b3304dc773607a: Status 404 returned error can't find the container with id dece5f18cd11aceb54139869897bedac4fe8a53b1667e4f357b3304dc773607a
Sep 17 18:55:50.326647 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:50.326562    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" event={"ID":"b94bb7b1-6c4e-4798-a971-44e9cdef7234","Type":"ContainerStarted","Data":"c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35"}
Sep 17 18:55:50.326647 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:50.326620    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" event={"ID":"b94bb7b1-6c4e-4798-a971-44e9cdef7234","Type":"ContainerStarted","Data":"dece5f18cd11aceb54139869897bedac4fe8a53b1667e4f357b3304dc773607a"}
Sep 17 18:55:50.327100 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:50.326650    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:55:50.358106 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:50.358048    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" podStartSLOduration=1.358026169 podStartE2EDuration="1.358026169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:50.357855756 +0000 UTC m=+971.217680416" watchObservedRunningTime="2026-09-17 18:55:50.358026169 +0000 UTC m=+971.217850818"
Sep 17 18:55:51.948277 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:55:51.948238    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" podUID="5e387c32-4684-4ea7-8e1f-5a07654ad261" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:56:01.947525 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:01.947484    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" podUID="5e387c32-4684-4ea7-8e1f-5a07654ad261" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:56:03.605572 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:03.605543    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:56:03.738383 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:03.738283    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e387c32-4684-4ea7-8e1f-5a07654ad261-tls-certs\") pod \"5e387c32-4684-4ea7-8e1f-5a07654ad261\" (UID: \"5e387c32-4684-4ea7-8e1f-5a07654ad261\") "
Sep 17 18:56:03.738564 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:03.738418    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vj5p\" (UniqueName: \"kubernetes.io/projected/5e387c32-4684-4ea7-8e1f-5a07654ad261-kube-api-access-6vj5p\") pod \"5e387c32-4684-4ea7-8e1f-5a07654ad261\" (UID: \"5e387c32-4684-4ea7-8e1f-5a07654ad261\") "
Sep 17 18:56:03.740650 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:03.740621    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e387c32-4684-4ea7-8e1f-5a07654ad261-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e387c32-4684-4ea7-8e1f-5a07654ad261" (UID: "5e387c32-4684-4ea7-8e1f-5a07654ad261"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:03.740761 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:03.740662    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e387c32-4684-4ea7-8e1f-5a07654ad261-kube-api-access-6vj5p" (OuterVolumeSpecName: "kube-api-access-6vj5p") pod "5e387c32-4684-4ea7-8e1f-5a07654ad261" (UID: "5e387c32-4684-4ea7-8e1f-5a07654ad261"). InnerVolumeSpecName "kube-api-access-6vj5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:03.839469 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:03.839433    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vj5p\" (UniqueName: \"kubernetes.io/projected/5e387c32-4684-4ea7-8e1f-5a07654ad261-kube-api-access-6vj5p\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:03.839469 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:03.839462    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e387c32-4684-4ea7-8e1f-5a07654ad261-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:04.406958 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:04.406923    2586 generic.go:356] "Generic (PLEG): container finished" podID="5e387c32-4684-4ea7-8e1f-5a07654ad261" containerID="03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc" exitCode=0
Sep 17 18:56:04.407138 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:04.406972    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"
Sep 17 18:56:04.407138 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:04.407011    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" event={"ID":"5e387c32-4684-4ea7-8e1f-5a07654ad261","Type":"ContainerDied","Data":"03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc"}
Sep 17 18:56:04.407138 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:04.407067    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk" event={"ID":"5e387c32-4684-4ea7-8e1f-5a07654ad261","Type":"ContainerDied","Data":"d400e9b8b6db8edd7c3b9298d1c15eb1f55ae4b2165a9b1debca4f6562dcf81b"}
Sep 17 18:56:04.407138 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:04.407095    2586 scope.go:117] "RemoveContainer" containerID="03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc"
Sep 17 18:56:04.417266 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:04.417248    2586 scope.go:117] "RemoveContainer" containerID="03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc"
Sep 17 18:56:04.417554 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:56:04.417537    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc\": container with ID starting with 03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc not found: ID does not exist" containerID="03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc"
Sep 17 18:56:04.417609 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:04.417563    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc"} err="failed to get container status \"03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc\": rpc error: code = NotFound desc = could not find container \"03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc\": container with ID starting with 03180f33e9ff054b6d741478859eb4d8fa43ae72ca470e36ca4cd185586903bc not found: ID does not exist"
Sep 17 18:56:04.439345 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:04.439315    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"]
Sep 17 18:56:04.449146 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:04.449114    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-zrsmk"]
Sep 17 18:56:05.458616 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:05.458578    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e387c32-4684-4ea7-8e1f-5a07654ad261" path="/var/lib/kubelet/pods/5e387c32-4684-4ea7-8e1f-5a07654ad261/volumes"
Sep 17 18:56:21.337170 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.337128    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:56:21.403626 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.403596    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"]
Sep 17 18:56:21.404032 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.404018    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e387c32-4684-4ea7-8e1f-5a07654ad261" containerName="main"
Sep 17 18:56:21.404096 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.404035    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e387c32-4684-4ea7-8e1f-5a07654ad261" containerName="main"
Sep 17 18:56:21.404132 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.404118    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e387c32-4684-4ea7-8e1f-5a07654ad261" containerName="main"
Sep 17 18:56:21.408691 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.408674    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.412247 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.412222    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 18:56:21.412605 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.412237    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 18:56:21.412605 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.412294    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-xcwkq\""
Sep 17 18:56:21.412605 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.412424    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 18:56:21.436871 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.436825    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"]
Sep 17 18:56:21.492728 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.492695    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.492959 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.492736    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.492959 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.492760    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.492959 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.492785    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.492959 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.492916    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.492959 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.492955    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtgc\" (UniqueName: \"kubernetes.io/projected/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-kube-api-access-rrtgc\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.593530 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.593430    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.593530 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.593474    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.593764 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.593529    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.593764 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.593557    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtgc\" (UniqueName: \"kubernetes.io/projected/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-kube-api-access-rrtgc\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.593764 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.593657    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.593764 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.593679    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.594041 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.593907    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.594041 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.593996    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.594203 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.594040    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.596116 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.596083    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.596332 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.596313    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.603576 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.603545    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtgc\" (UniqueName: \"kubernetes.io/projected/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-kube-api-access-rrtgc\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:21.721332 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:21.721299    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:22.012504 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:22.012473    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"]
Sep 17 18:56:22.013263 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:56:22.013237    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a0d2e0_4ccd_4a94_9997_b6d9fdd7f6c2.slice/crio-2b5171f7b3401f69dcc3e45bcb466c49ec22a7057af1df5b25cfa1c09f1d964d WatchSource:0}: Error finding container 2b5171f7b3401f69dcc3e45bcb466c49ec22a7057af1df5b25cfa1c09f1d964d: Status 404 returned error can't find the container with id 2b5171f7b3401f69dcc3e45bcb466c49ec22a7057af1df5b25cfa1c09f1d964d
Sep 17 18:56:22.509413 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:22.509376    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx" event={"ID":"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2","Type":"ContainerStarted","Data":"df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7"}
Sep 17 18:56:22.509822 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:22.509435    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx" event={"ID":"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2","Type":"ContainerStarted","Data":"2b5171f7b3401f69dcc3e45bcb466c49ec22a7057af1df5b25cfa1c09f1d964d"}
Sep 17 18:56:22.529406 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:22.529358    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx" podStartSLOduration=1.5293413340000002 podStartE2EDuration="1.529341334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:22.528355516 +0000 UTC m=+1003.388180177" watchObservedRunningTime="2026-09-17 18:56:22.529341334 +0000 UTC m=+1003.389165982"
Sep 17 18:56:22.722072 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:22.722043    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:28.615261 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:28.615226    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"]
Sep 17 18:56:28.625537 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:56:28.625499    2586 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:56:28.625537 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:28.625537 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:28.625537 ip-10-0-129-204 kubenswrapper[2586]:  > containerID="df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7" cmd=["/bin/sleep","15"]
Sep 17 18:56:28.625915 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:28.625583    2586 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx" message="" err=<
Sep 17 18:56:28.625915 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:28.625915 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:28.625915 ip-10-0-129-204 kubenswrapper[2586]:  >
Sep 17 18:56:28.625915 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:56:28.625604    2586 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:56:28.625915 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:28.625915 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:28.625915 ip-10-0-129-204 kubenswrapper[2586]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx" podUID="a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" containerName="main" containerID="cri-o://df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7"
Sep 17 18:56:28.625915 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:28.625698    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx" podUID="a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" containerName="main" containerID="cri-o://df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7" gracePeriod=60
Sep 17 18:56:28.978086 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:28.978061    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:29.168146 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.168056    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tls-certs\") pod \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") "
Sep 17 18:56:29.168146 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.168101    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrtgc\" (UniqueName: \"kubernetes.io/projected/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-kube-api-access-rrtgc\") pod \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") "
Sep 17 18:56:29.168402 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.168180    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-dshm\") pod \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") "
Sep 17 18:56:29.168402 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.168216    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tmp-dir\") pod \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") "
Sep 17 18:56:29.168402 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.168246    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-model-cache\") pod \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") "
Sep 17 18:56:29.168402 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.168272    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-home\") pod \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\" (UID: \"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2\") "
Sep 17 18:56:29.168626 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.168509    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" (UID: "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:29.168626 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.168537    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" (UID: "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:29.168744 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.168623    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-home" (OuterVolumeSpecName: "home") pod "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" (UID: "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:29.170623 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.170596    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-kube-api-access-rrtgc" (OuterVolumeSpecName: "kube-api-access-rrtgc") pod "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" (UID: "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2"). InnerVolumeSpecName "kube-api-access-rrtgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:29.170623 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.170611    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-dshm" (OuterVolumeSpecName: "dshm") pod "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" (UID: "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:29.170773 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.170647    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" (UID: "a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:29.269087 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.269054    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:29.269087 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.269085    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrtgc\" (UniqueName: \"kubernetes.io/projected/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-kube-api-access-rrtgc\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:29.269087 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.269094    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:29.269340 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.269105    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:29.269340 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.269116    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:29.269340 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.269124    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:29.556663 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.556633    2586 generic.go:356] "Generic (PLEG): container finished" podID="a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" containerID="df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7" exitCode=0
Sep 17 18:56:29.556868 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.556677    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"
Sep 17 18:56:29.556868 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.556722    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx" event={"ID":"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2","Type":"ContainerDied","Data":"df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7"}
Sep 17 18:56:29.556868 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.556777    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx" event={"ID":"a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2","Type":"ContainerDied","Data":"2b5171f7b3401f69dcc3e45bcb466c49ec22a7057af1df5b25cfa1c09f1d964d"}
Sep 17 18:56:29.556868 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.556798    2586 scope.go:117] "RemoveContainer" containerID="df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7"
Sep 17 18:56:29.566129 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.566093    2586 scope.go:117] "RemoveContainer" containerID="df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7"
Sep 17 18:56:29.566462 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:56:29.566439    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7\": container with ID starting with df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7 not found: ID does not exist" containerID="df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7"
Sep 17 18:56:29.566534 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.566470    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7"} err="failed to get container status \"df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7\": rpc error: code = NotFound desc = could not find container \"df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7\": container with ID starting with df5fb44a7fe0d55a4e89416e2f5182f32cfbe59964a95ac341ec11f575ffc2b7 not found: ID does not exist"
Sep 17 18:56:29.574571 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.574540    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"]
Sep 17 18:56:29.580993 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:29.580969    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xzldx"]
Sep 17 18:56:31.465256 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:31.465223    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" path="/var/lib/kubelet/pods/a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2/volumes"
Sep 17 18:56:39.091611 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.091569    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"]
Sep 17 18:56:39.092043 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.092006    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" containerName="main"
Sep 17 18:56:39.092043 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.092021    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" containerName="main"
Sep 17 18:56:39.092144 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.092095    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6a0d2e0-4ccd-4a94-9997-b6d9fdd7f6c2" containerName="main"
Sep 17 18:56:39.096266 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.096251    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.100183 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.100155    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 18:56:39.100338 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.100242    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-vtgq9\""
Sep 17 18:56:39.100338 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.100254    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 18:56:39.100338 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.100271    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 18:56:39.105464 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.105441    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"]
Sep 17 18:56:39.137338 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.137297    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.137520 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.137362    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.137520 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.137395    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.137520 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.137471    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.137657 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.137524    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5sv\" (UniqueName: \"kubernetes.io/projected/52c8c584-4885-48e7-9f44-9ef9eb7a8670-kube-api-access-nl5sv\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.137657 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.137550    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.238914 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.238872    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5sv\" (UniqueName: \"kubernetes.io/projected/52c8c584-4885-48e7-9f44-9ef9eb7a8670-kube-api-access-nl5sv\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.239119 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.238919    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.239119 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.238988    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.239275 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.239121    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.239275 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.239168    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.239275 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.239228    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.239441 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.239422    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.239501 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.239443    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.239573 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.239552    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.241550 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.241528    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.241847 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.241813    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.247740 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.247721    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5sv\" (UniqueName: \"kubernetes.io/projected/52c8c584-4885-48e7-9f44-9ef9eb7a8670-kube-api-access-nl5sv\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.407481 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.407391    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:39.684726 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:39.684702    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"]
Sep 17 18:56:39.686804 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:56:39.686771    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c8c584_4885_48e7_9f44_9ef9eb7a8670.slice/crio-300e12fec07d5519cdd85b8bbb02845b71d01a246f9d42f2d5ca23ba704ad260 WatchSource:0}: Error finding container 300e12fec07d5519cdd85b8bbb02845b71d01a246f9d42f2d5ca23ba704ad260: Status 404 returned error can't find the container with id 300e12fec07d5519cdd85b8bbb02845b71d01a246f9d42f2d5ca23ba704ad260
Sep 17 18:56:40.619381 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:40.619351    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr" event={"ID":"52c8c584-4885-48e7-9f44-9ef9eb7a8670","Type":"ContainerStarted","Data":"c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde"}
Sep 17 18:56:40.619744 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:40.619399    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr" event={"ID":"52c8c584-4885-48e7-9f44-9ef9eb7a8670","Type":"ContainerStarted","Data":"300e12fec07d5519cdd85b8bbb02845b71d01a246f9d42f2d5ca23ba704ad260"}
Sep 17 18:56:40.640550 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:40.640497    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr" podStartSLOduration=1.640479999 podStartE2EDuration="1.640479999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:40.637820748 +0000 UTC m=+1021.497645415" watchObservedRunningTime="2026-09-17 18:56:40.640479999 +0000 UTC m=+1021.500304646"
Sep 17 18:56:41.407778 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:41.407739    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:44.633010 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:44.632974    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"]
Sep 17 18:56:44.642432 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:56:44.642394    2586 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:56:44.642432 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:44.642432 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:44.642432 ip-10-0-129-204 kubenswrapper[2586]:  > containerID="c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde" cmd=["/bin/sleep","15"]
Sep 17 18:56:44.642687 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:44.642442    2586 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr" message="" err=<
Sep 17 18:56:44.642687 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:44.642687 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:44.642687 ip-10-0-129-204 kubenswrapper[2586]:  >
Sep 17 18:56:44.642687 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:56:44.642456    2586 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:56:44.642687 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:44.642687 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:44.642687 ip-10-0-129-204 kubenswrapper[2586]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr" podUID="52c8c584-4885-48e7-9f44-9ef9eb7a8670" containerName="main" containerID="cri-o://c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde"
Sep 17 18:56:44.642687 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:44.642497    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr" podUID="52c8c584-4885-48e7-9f44-9ef9eb7a8670" containerName="main" containerID="cri-o://c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde" gracePeriod=60
Sep 17 18:56:44.998864 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:44.998826    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:45.089415 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.089363    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-home\") pod \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") "
Sep 17 18:56:45.089415 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.089426    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tmp-dir\") pod \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") "
Sep 17 18:56:45.089649 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.089601    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tls-certs\") pod \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") "
Sep 17 18:56:45.089701 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.089650    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52c8c584-4885-48e7-9f44-9ef9eb7a8670" (UID: "52c8c584-4885-48e7-9f44-9ef9eb7a8670"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:45.089701 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.089668    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-home" (OuterVolumeSpecName: "home") pod "52c8c584-4885-48e7-9f44-9ef9eb7a8670" (UID: "52c8c584-4885-48e7-9f44-9ef9eb7a8670"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:45.089701 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.089678    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl5sv\" (UniqueName: \"kubernetes.io/projected/52c8c584-4885-48e7-9f44-9ef9eb7a8670-kube-api-access-nl5sv\") pod \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") "
Sep 17 18:56:45.089871 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.089705    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-model-cache\") pod \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") "
Sep 17 18:56:45.089871 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.089723    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-dshm\") pod \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\" (UID: \"52c8c584-4885-48e7-9f44-9ef9eb7a8670\") "
Sep 17 18:56:45.090007 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.089989    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:45.090062 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.090020    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:45.090062 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.089986    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-model-cache" (OuterVolumeSpecName: "model-cache") pod "52c8c584-4885-48e7-9f44-9ef9eb7a8670" (UID: "52c8c584-4885-48e7-9f44-9ef9eb7a8670"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:45.092010 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.091982    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-dshm" (OuterVolumeSpecName: "dshm") pod "52c8c584-4885-48e7-9f44-9ef9eb7a8670" (UID: "52c8c584-4885-48e7-9f44-9ef9eb7a8670"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:45.092111 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.092055    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52c8c584-4885-48e7-9f44-9ef9eb7a8670" (UID: "52c8c584-4885-48e7-9f44-9ef9eb7a8670"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:45.092111 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.092056    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c8c584-4885-48e7-9f44-9ef9eb7a8670-kube-api-access-nl5sv" (OuterVolumeSpecName: "kube-api-access-nl5sv") pod "52c8c584-4885-48e7-9f44-9ef9eb7a8670" (UID: "52c8c584-4885-48e7-9f44-9ef9eb7a8670"). InnerVolumeSpecName "kube-api-access-nl5sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:45.190789 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.190703    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c8c584-4885-48e7-9f44-9ef9eb7a8670-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:45.190789 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.190731    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl5sv\" (UniqueName: \"kubernetes.io/projected/52c8c584-4885-48e7-9f44-9ef9eb7a8670-kube-api-access-nl5sv\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:45.190789 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.190741    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:45.190789 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.190751    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52c8c584-4885-48e7-9f44-9ef9eb7a8670-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:56:45.647122 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.647086    2586 generic.go:356] "Generic (PLEG): container finished" podID="52c8c584-4885-48e7-9f44-9ef9eb7a8670" containerID="c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde" exitCode=0
Sep 17 18:56:45.647623 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.647137    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"
Sep 17 18:56:45.647623 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.647144    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr" event={"ID":"52c8c584-4885-48e7-9f44-9ef9eb7a8670","Type":"ContainerDied","Data":"c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde"}
Sep 17 18:56:45.647623 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.647249    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr" event={"ID":"52c8c584-4885-48e7-9f44-9ef9eb7a8670","Type":"ContainerDied","Data":"300e12fec07d5519cdd85b8bbb02845b71d01a246f9d42f2d5ca23ba704ad260"}
Sep 17 18:56:45.647623 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.647271    2586 scope.go:117] "RemoveContainer" containerID="c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde"
Sep 17 18:56:45.656675 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.656650    2586 scope.go:117] "RemoveContainer" containerID="c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde"
Sep 17 18:56:45.657047 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:56:45.657022    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde\": container with ID starting with c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde not found: ID does not exist" containerID="c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde"
Sep 17 18:56:45.657111 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.657056    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde"} err="failed to get container status \"c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde\": rpc error: code = NotFound desc = could not find container \"c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde\": container with ID starting with c6b49e93b43b9c2a68a6fbe54ce4e2888ef94f245e82a449aff3ba54f1bd8bde not found: ID does not exist"
Sep 17 18:56:45.669845 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.669801    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"]
Sep 17 18:56:45.675257 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:45.675230    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjf2vr"]
Sep 17 18:56:47.459663 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:47.459628    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c8c584-4885-48e7-9f44-9ef9eb7a8670" path="/var/lib/kubelet/pods/52c8c584-4885-48e7-9f44-9ef9eb7a8670/volumes"
Sep 17 18:56:58.051934 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.051897    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"]
Sep 17 18:56:58.052366 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.052302    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c8c584-4885-48e7-9f44-9ef9eb7a8670" containerName="main"
Sep 17 18:56:58.052366 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.052314    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c8c584-4885-48e7-9f44-9ef9eb7a8670" containerName="main"
Sep 17 18:56:58.052440 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.052406    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c8c584-4885-48e7-9f44-9ef9eb7a8670" containerName="main"
Sep 17 18:56:58.055462 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.055444    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 18:56:58.060704 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.060669    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 18:56:58.061770 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.061747    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 18:56:58.061908 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.061747    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-wftxp\""
Sep 17 18:56:58.061908 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.061747    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 18:56:58.061908 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.061814    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-8xkdm\""
Sep 17 18:56:58.078051 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.078014    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"]
Sep 17 18:56:58.107512 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.107457    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tzd\" (UniqueName: \"kubernetes.io/projected/a0c05053-774c-4238-b05a-29223910e875-kube-api-access-p4tzd\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn\" (UID: \"a0c05053-774c-4238-b05a-29223910e875\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 18:56:58.107683 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.107515    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c05053-774c-4238-b05a-29223910e875-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn\" (UID: \"a0c05053-774c-4238-b05a-29223910e875\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 18:56:58.208530 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.208468    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tzd\" (UniqueName: \"kubernetes.io/projected/a0c05053-774c-4238-b05a-29223910e875-kube-api-access-p4tzd\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn\" (UID: \"a0c05053-774c-4238-b05a-29223910e875\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 18:56:58.208530 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.208533    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c05053-774c-4238-b05a-29223910e875-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn\" (UID: \"a0c05053-774c-4238-b05a-29223910e875\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 18:56:58.211400 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.211363    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c05053-774c-4238-b05a-29223910e875-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn\" (UID: \"a0c05053-774c-4238-b05a-29223910e875\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 18:56:58.221269 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.221243    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tzd\" (UniqueName: \"kubernetes.io/projected/a0c05053-774c-4238-b05a-29223910e875-kube-api-access-p4tzd\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn\" (UID: \"a0c05053-774c-4238-b05a-29223910e875\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 18:56:58.369987 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.369941    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 18:56:58.683075 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.683044    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"]
Sep 17 18:56:58.683493 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:56:58.683466    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c05053_774c_4238_b05a_29223910e875.slice/crio-75a4ab03db2f1cd0e0e69d731cea9263dcd0193f057a86a4d08a7690e9aab5ff WatchSource:0}: Error finding container 75a4ab03db2f1cd0e0e69d731cea9263dcd0193f057a86a4d08a7690e9aab5ff: Status 404 returned error can't find the container with id 75a4ab03db2f1cd0e0e69d731cea9263dcd0193f057a86a4d08a7690e9aab5ff
Sep 17 18:56:58.719145 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:58.719105    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn" event={"ID":"a0c05053-774c-4238-b05a-29223910e875","Type":"ContainerStarted","Data":"75a4ab03db2f1cd0e0e69d731cea9263dcd0193f057a86a4d08a7690e9aab5ff"}
Sep 17 18:56:59.726499 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:59.726403    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn" event={"ID":"a0c05053-774c-4238-b05a-29223910e875","Type":"ContainerStarted","Data":"24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853"}
Sep 17 18:56:59.726499 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:59.726475    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 18:56:59.748597 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:56:59.748539    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn" podStartSLOduration=1.7485215219999999 podStartE2EDuration="1.748521522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:59.746684193 +0000 UTC m=+1040.606508861" watchObservedRunningTime="2026-09-17 18:56:59.748521522 +0000 UTC m=+1040.608346195"
Sep 17 18:57:30.733628 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:57:30.733548    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 18:57:36.655124 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:57:36.655078    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"]
Sep 17 18:57:41.336527 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:57:41.336477    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" podUID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:57:51.336939 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:57:51.336900    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" podUID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:57:51.674822 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:57:51.674734    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" podUID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" containerName="main" containerID="cri-o://c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35" gracePeriod=45
Sep 17 18:58:01.336902 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:01.336829    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" podUID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:58:01.337343 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:01.336991    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:58:11.336929 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:11.336894    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" podUID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:58:21.337042 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:21.337004    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" podUID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:58:21.928710 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:21.928684    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:58:21.940788 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:21.940716    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9b2n\" (UniqueName: \"kubernetes.io/projected/b94bb7b1-6c4e-4798-a971-44e9cdef7234-kube-api-access-w9b2n\") pod \"b94bb7b1-6c4e-4798-a971-44e9cdef7234\" (UID: \"b94bb7b1-6c4e-4798-a971-44e9cdef7234\") "
Sep 17 18:58:21.940951 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:21.940809    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b94bb7b1-6c4e-4798-a971-44e9cdef7234-tls-certs\") pod \"b94bb7b1-6c4e-4798-a971-44e9cdef7234\" (UID: \"b94bb7b1-6c4e-4798-a971-44e9cdef7234\") "
Sep 17 18:58:21.943120 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:21.943096    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b94bb7b1-6c4e-4798-a971-44e9cdef7234-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b94bb7b1-6c4e-4798-a971-44e9cdef7234" (UID: "b94bb7b1-6c4e-4798-a971-44e9cdef7234"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:58:21.943216 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:21.943137    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94bb7b1-6c4e-4798-a971-44e9cdef7234-kube-api-access-w9b2n" (OuterVolumeSpecName: "kube-api-access-w9b2n") pod "b94bb7b1-6c4e-4798-a971-44e9cdef7234" (UID: "b94bb7b1-6c4e-4798-a971-44e9cdef7234"). InnerVolumeSpecName "kube-api-access-w9b2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:58:22.041545 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.041514    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b94bb7b1-6c4e-4798-a971-44e9cdef7234-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:58:22.041545 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.041540    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9b2n\" (UniqueName: \"kubernetes.io/projected/b94bb7b1-6c4e-4798-a971-44e9cdef7234-kube-api-access-w9b2n\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 18:58:22.199055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.198952    2586 generic.go:356] "Generic (PLEG): container finished" podID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" containerID="c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35" exitCode=0
Sep 17 18:58:22.199055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.198995    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"
Sep 17 18:58:22.199055 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.199013    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" event={"ID":"b94bb7b1-6c4e-4798-a971-44e9cdef7234","Type":"ContainerDied","Data":"c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35"}
Sep 17 18:58:22.199322 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.199065    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h" event={"ID":"b94bb7b1-6c4e-4798-a971-44e9cdef7234","Type":"ContainerDied","Data":"dece5f18cd11aceb54139869897bedac4fe8a53b1667e4f357b3304dc773607a"}
Sep 17 18:58:22.199322 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.199091    2586 scope.go:117] "RemoveContainer" containerID="c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35"
Sep 17 18:58:22.209152 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.209135    2586 scope.go:117] "RemoveContainer" containerID="c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35"
Sep 17 18:58:22.209414 ip-10-0-129-204 kubenswrapper[2586]: E0917 18:58:22.209400    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35\": container with ID starting with c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35 not found: ID does not exist" containerID="c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35"
Sep 17 18:58:22.209463 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.209421    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35"} err="failed to get container status \"c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35\": rpc error: code = NotFound desc = could not find container \"c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35\": container with ID starting with c633463c18e68eb2c0abbb28b21fd86274772142beb874027eeb4a8d7877fa35 not found: ID does not exist"
Sep 17 18:58:22.232697 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.232666    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"]
Sep 17 18:58:22.238615 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:22.238593    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75g769h"]
Sep 17 18:58:23.459965 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:23.459929    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" path="/var/lib/kubelet/pods/b94bb7b1-6c4e-4798-a971-44e9cdef7234/volumes"
Sep 17 18:58:46.290193 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.290157    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"]
Sep 17 18:58:46.290756 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.290726    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" containerName="main"
Sep 17 18:58:46.290756 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.290746    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" containerName="main"
Sep 17 18:58:46.290911 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.290872    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94bb7b1-6c4e-4798-a971-44e9cdef7234" containerName="main"
Sep 17 18:58:46.301957 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.301924    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:58:46.308371 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.308341    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 18:58:46.309030 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.308378    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 18:58:46.309163 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.308378    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-t76vb\""
Sep 17 18:58:46.312909 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.312865    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-v9dfn\""
Sep 17 18:58:46.312909 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.312909    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 18:58:46.315305 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.315275    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"]
Sep 17 18:58:46.337223 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.337186    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25t8p\" (UniqueName: \"kubernetes.io/projected/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-kube-api-access-25t8p\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxbm54\" (UID: \"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:58:46.337418 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.337293    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxbm54\" (UID: \"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:58:46.437765 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.437721    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxbm54\" (UID: \"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:58:46.437996 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.437803    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25t8p\" (UniqueName: \"kubernetes.io/projected/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-kube-api-access-25t8p\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxbm54\" (UID: \"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:58:46.440561 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.440540    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxbm54\" (UID: \"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:58:46.448705 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.448680    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25t8p\" (UniqueName: \"kubernetes.io/projected/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-kube-api-access-25t8p\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxbm54\" (UID: \"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:58:46.612603 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.612575    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:58:46.888677 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:46.888649    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"]
Sep 17 18:58:46.890166 ip-10-0-129-204 kubenswrapper[2586]: W0917 18:58:46.890131    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1f6c8e_6c57_4d2f_92ab_3815b69b9b04.slice/crio-80c0603d46941b02857ca802657494679f44e56f7a69fdf4d75654d4bfbc610a WatchSource:0}: Error finding container 80c0603d46941b02857ca802657494679f44e56f7a69fdf4d75654d4bfbc610a: Status 404 returned error can't find the container with id 80c0603d46941b02857ca802657494679f44e56f7a69fdf4d75654d4bfbc610a
Sep 17 18:58:47.341912 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:47.341873    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54" event={"ID":"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04","Type":"ContainerStarted","Data":"511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07"}
Sep 17 18:58:47.342372 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:47.341937    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:58:47.342372 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:47.341954    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54" event={"ID":"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04","Type":"ContainerStarted","Data":"80c0603d46941b02857ca802657494679f44e56f7a69fdf4d75654d4bfbc610a"}
Sep 17 18:58:47.380865 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:58:47.380792    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54" podStartSLOduration=1.380773839 podStartE2EDuration="1.380773839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:47.379495432 +0000 UTC m=+1148.239320102" watchObservedRunningTime="2026-09-17 18:58:47.380773839 +0000 UTC m=+1148.240598487"
Sep 17 18:59:18.350568 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:18.350537    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:59:21.318235 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:21.318193    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"]
Sep 17 18:59:28.350277 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:28.350230    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54" podUID="4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:59:33.834921 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:33.834880    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"]
Sep 17 18:59:36.338771 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:36.338699    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54" podUID="4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" containerName="main" containerID="cri-o://511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07" gracePeriod=45
Sep 17 18:59:38.349951 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:38.349914    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54" podUID="4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:59:40.734063 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:40.734024    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn" podUID="a0c05053-774c-4238-b05a-29223910e875" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:59:41.425623 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:41.425596    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:59:41.429646 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:41.429619    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 18:59:48.350744 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:48.350704    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54" podUID="4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:59:48.351174 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:48.350810    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 18:59:48.854542 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:48.854475    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn" podUID="a0c05053-774c-4238-b05a-29223910e875" containerName="main" containerID="cri-o://24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853" gracePeriod=45
Sep 17 18:59:50.733327 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:50.733284    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn" podUID="a0c05053-774c-4238-b05a-29223910e875" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:59:58.349944 ip-10-0-129-204 kubenswrapper[2586]: I0917 18:59:58.349903    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54" podUID="4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:00:00.733645 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:00.733609    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn" podUID="a0c05053-774c-4238-b05a-29223910e875" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:00:00.734157 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:00.733739    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 19:00:06.575974 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.575949    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 19:00:06.728863 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.728760    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25t8p\" (UniqueName: \"kubernetes.io/projected/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-kube-api-access-25t8p\") pod \"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04\" (UID: \"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04\") "
Sep 17 19:00:06.728863 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.728794    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-tls-certs\") pod \"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04\" (UID: \"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04\") "
Sep 17 19:00:06.731160 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.731130    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-kube-api-access-25t8p" (OuterVolumeSpecName: "kube-api-access-25t8p") pod "4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" (UID: "4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04"). InnerVolumeSpecName "kube-api-access-25t8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:00:06.731329 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.731305    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" (UID: "4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:00:06.791563 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.791532    2586 generic.go:356] "Generic (PLEG): container finished" podID="4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" containerID="511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07" exitCode=0
Sep 17 19:00:06.791748 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.791570    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54" event={"ID":"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04","Type":"ContainerDied","Data":"511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07"}
Sep 17 19:00:06.791748 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.791575    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"
Sep 17 19:00:06.791748 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.791611    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54" event={"ID":"4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04","Type":"ContainerDied","Data":"80c0603d46941b02857ca802657494679f44e56f7a69fdf4d75654d4bfbc610a"}
Sep 17 19:00:06.791748 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.791633    2586 scope.go:117] "RemoveContainer" containerID="511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07"
Sep 17 19:00:06.801211 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.801192    2586 scope.go:117] "RemoveContainer" containerID="511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07"
Sep 17 19:00:06.801491 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:00:06.801472    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07\": container with ID starting with 511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07 not found: ID does not exist" containerID="511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07"
Sep 17 19:00:06.801551 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.801501    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07"} err="failed to get container status \"511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07\": rpc error: code = NotFound desc = could not find container \"511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07\": container with ID starting with 511af32d854b7f52e69be2c9b5e81479d568904665ebe452794fd062b22d3d07 not found: ID does not exist"
Sep 17 19:00:06.819762 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.819733    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"]
Sep 17 19:00:06.825423 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.825397    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedxbm54"]
Sep 17 19:00:06.829451 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.829423    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25t8p\" (UniqueName: \"kubernetes.io/projected/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-kube-api-access-25t8p\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:00:06.829451 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:06.829446    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:00:07.467801 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:07.467767    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" path="/var/lib/kubelet/pods/4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04/volumes"
Sep 17 19:00:10.734217 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:10.734177    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn" podUID="a0c05053-774c-4238-b05a-29223910e875" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:00:19.103390 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.103361    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 19:00:19.132801 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.132772    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c05053-774c-4238-b05a-29223910e875-tls-certs\") pod \"a0c05053-774c-4238-b05a-29223910e875\" (UID: \"a0c05053-774c-4238-b05a-29223910e875\") "
Sep 17 19:00:19.133003 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.132828    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4tzd\" (UniqueName: \"kubernetes.io/projected/a0c05053-774c-4238-b05a-29223910e875-kube-api-access-p4tzd\") pod \"a0c05053-774c-4238-b05a-29223910e875\" (UID: \"a0c05053-774c-4238-b05a-29223910e875\") "
Sep 17 19:00:19.135167 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.135140    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c05053-774c-4238-b05a-29223910e875-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0c05053-774c-4238-b05a-29223910e875" (UID: "a0c05053-774c-4238-b05a-29223910e875"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:00:19.135273 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.135189    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c05053-774c-4238-b05a-29223910e875-kube-api-access-p4tzd" (OuterVolumeSpecName: "kube-api-access-p4tzd") pod "a0c05053-774c-4238-b05a-29223910e875" (UID: "a0c05053-774c-4238-b05a-29223910e875"). InnerVolumeSpecName "kube-api-access-p4tzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:00:19.234054 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.233948    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c05053-774c-4238-b05a-29223910e875-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:00:19.234054 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.233996    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tzd\" (UniqueName: \"kubernetes.io/projected/a0c05053-774c-4238-b05a-29223910e875-kube-api-access-p4tzd\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:00:19.868913 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.868861    2586 generic.go:356] "Generic (PLEG): container finished" podID="a0c05053-774c-4238-b05a-29223910e875" containerID="24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853" exitCode=0
Sep 17 19:00:19.869131 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.868938    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"
Sep 17 19:00:19.869131 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.868979    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn" event={"ID":"a0c05053-774c-4238-b05a-29223910e875","Type":"ContainerDied","Data":"24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853"}
Sep 17 19:00:19.869131 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.869092    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn" event={"ID":"a0c05053-774c-4238-b05a-29223910e875","Type":"ContainerDied","Data":"75a4ab03db2f1cd0e0e69d731cea9263dcd0193f057a86a4d08a7690e9aab5ff"}
Sep 17 19:00:19.869131 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.869127    2586 scope.go:117] "RemoveContainer" containerID="24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853"
Sep 17 19:00:19.878749 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.878733    2586 scope.go:117] "RemoveContainer" containerID="24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853"
Sep 17 19:00:19.879037 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:00:19.879019    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853\": container with ID starting with 24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853 not found: ID does not exist" containerID="24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853"
Sep 17 19:00:19.879123 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.879044    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853"} err="failed to get container status \"24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853\": rpc error: code = NotFound desc = could not find container \"24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853\": container with ID starting with 24a5fd2df4535627ad103da2f85b168dee6c45dd75629d3d5294e82a92d82853 not found: ID does not exist"
Sep 17 19:00:19.887754 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.887730    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"]
Sep 17 19:00:19.891757 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:19.891731    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2bjjn"]
Sep 17 19:00:21.459275 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:21.459233    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0c05053-774c-4238-b05a-29223910e875" path="/var/lib/kubelet/pods/a0c05053-774c-4238-b05a-29223910e875/volumes"
Sep 17 19:00:38.903658 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.903615    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"]
Sep 17 19:00:38.904353 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.904034    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c05053-774c-4238-b05a-29223910e875" containerName="main"
Sep 17 19:00:38.904353 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.904050    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c05053-774c-4238-b05a-29223910e875" containerName="main"
Sep 17 19:00:38.904353 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.904060    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" containerName="main"
Sep 17 19:00:38.904353 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.904065    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" containerName="main"
Sep 17 19:00:38.904353 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.904132    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c05053-774c-4238-b05a-29223910e875" containerName="main"
Sep 17 19:00:38.904353 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.904144    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1f6c8e-6c57-4d2f-92ab-3815b69b9b04" containerName="main"
Sep 17 19:00:38.907115 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.907097    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:00:38.911096 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.911072    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 19:00:38.911233 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.911075    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 19:00:38.911233 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.911083    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 19:00:38.911233 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.911138    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-tvcrb\""
Sep 17 19:00:38.911233 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.911118    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-hrxhq\""
Sep 17 19:00:38.925352 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.925326    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"]
Sep 17 19:00:38.996824 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.996786    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xltrr\" (UniqueName: \"kubernetes.io/projected/139dcb7e-43da-4b63-b1ec-494029f24309-kube-api-access-xltrr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c\" (UID: \"139dcb7e-43da-4b63-b1ec-494029f24309\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:00:38.996824 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:38.996827    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139dcb7e-43da-4b63-b1ec-494029f24309-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c\" (UID: \"139dcb7e-43da-4b63-b1ec-494029f24309\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:00:39.098193 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:39.098153    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xltrr\" (UniqueName: \"kubernetes.io/projected/139dcb7e-43da-4b63-b1ec-494029f24309-kube-api-access-xltrr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c\" (UID: \"139dcb7e-43da-4b63-b1ec-494029f24309\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:00:39.098193 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:39.098196    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139dcb7e-43da-4b63-b1ec-494029f24309-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c\" (UID: \"139dcb7e-43da-4b63-b1ec-494029f24309\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:00:39.100975 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:39.100954    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139dcb7e-43da-4b63-b1ec-494029f24309-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c\" (UID: \"139dcb7e-43da-4b63-b1ec-494029f24309\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:00:39.108274 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:39.108251    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xltrr\" (UniqueName: \"kubernetes.io/projected/139dcb7e-43da-4b63-b1ec-494029f24309-kube-api-access-xltrr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c\" (UID: \"139dcb7e-43da-4b63-b1ec-494029f24309\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:00:39.218234 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:39.218137    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:00:39.492474 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:39.492443    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"]
Sep 17 19:00:39.493933 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:00:39.493905    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139dcb7e_43da_4b63_b1ec_494029f24309.slice/crio-3ba0bd506209da9653efbc1d395bc71ab858582e85aa1f2c34d3c15f259cdf8b WatchSource:0}: Error finding container 3ba0bd506209da9653efbc1d395bc71ab858582e85aa1f2c34d3c15f259cdf8b: Status 404 returned error can't find the container with id 3ba0bd506209da9653efbc1d395bc71ab858582e85aa1f2c34d3c15f259cdf8b
Sep 17 19:00:39.988402 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:39.988355    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c" event={"ID":"139dcb7e-43da-4b63-b1ec-494029f24309","Type":"ContainerStarted","Data":"bb379ac3afc366b815c2999b45363409c55d1d1b9aabf215d6c8b254a379e3d5"}
Sep 17 19:00:39.988889 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:39.988456    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c" event={"ID":"139dcb7e-43da-4b63-b1ec-494029f24309","Type":"ContainerStarted","Data":"3ba0bd506209da9653efbc1d395bc71ab858582e85aa1f2c34d3c15f259cdf8b"}
Sep 17 19:00:39.988889 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:39.988507    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:00:40.015003 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:00:40.014952    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c" podStartSLOduration=2.014936345 podStartE2EDuration="2.014936345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:00:40.012422134 +0000 UTC m=+1260.872246803" watchObservedRunningTime="2026-09-17 19:00:40.014936345 +0000 UTC m=+1260.874760993"
Sep 17 19:01:03.512900 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.512850    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"]
Sep 17 19:01:03.516473 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.516451    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:01:03.519716 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.519694    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-8xphn\""
Sep 17 19:01:03.520796 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.520777    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 19:01:03.520971 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.520953    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 19:01:03.521028 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.521008    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-zkxn8\""
Sep 17 19:01:03.521080 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.520954    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 19:01:03.527147 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.527126    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"]
Sep 17 19:01:03.614244 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.614205    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eed5df7-e9bd-47a8-b68a-9920bc484335-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq\" (UID: \"9eed5df7-e9bd-47a8-b68a-9920bc484335\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:01:03.614244 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.614247    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9598x\" (UniqueName: \"kubernetes.io/projected/9eed5df7-e9bd-47a8-b68a-9920bc484335-kube-api-access-9598x\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq\" (UID: \"9eed5df7-e9bd-47a8-b68a-9920bc484335\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:01:03.714774 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.714733    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eed5df7-e9bd-47a8-b68a-9920bc484335-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq\" (UID: \"9eed5df7-e9bd-47a8-b68a-9920bc484335\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:01:03.714991 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.714779    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9598x\" (UniqueName: \"kubernetes.io/projected/9eed5df7-e9bd-47a8-b68a-9920bc484335-kube-api-access-9598x\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq\" (UID: \"9eed5df7-e9bd-47a8-b68a-9920bc484335\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:01:03.717577 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.717547    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eed5df7-e9bd-47a8-b68a-9920bc484335-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq\" (UID: \"9eed5df7-e9bd-47a8-b68a-9920bc484335\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:01:03.726381 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.726352    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9598x\" (UniqueName: \"kubernetes.io/projected/9eed5df7-e9bd-47a8-b68a-9920bc484335-kube-api-access-9598x\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq\" (UID: \"9eed5df7-e9bd-47a8-b68a-9920bc484335\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:01:03.827519 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:03.827488    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:01:04.105965 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:04.105934    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"]
Sep 17 19:01:04.107794 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:01:04.107760    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eed5df7_e9bd_47a8_b68a_9920bc484335.slice/crio-b2cc1030b51ec85471653308c2c145a68c19af3dac0cb585a5ce95f02856557e WatchSource:0}: Error finding container b2cc1030b51ec85471653308c2c145a68c19af3dac0cb585a5ce95f02856557e: Status 404 returned error can't find the container with id b2cc1030b51ec85471653308c2c145a68c19af3dac0cb585a5ce95f02856557e
Sep 17 19:01:04.126743 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:04.126705    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq" event={"ID":"9eed5df7-e9bd-47a8-b68a-9920bc484335","Type":"ContainerStarted","Data":"b2cc1030b51ec85471653308c2c145a68c19af3dac0cb585a5ce95f02856557e"}
Sep 17 19:01:05.134605 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:05.134562    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq" event={"ID":"9eed5df7-e9bd-47a8-b68a-9920bc484335","Type":"ContainerStarted","Data":"b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb"}
Sep 17 19:01:05.135074 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:05.134767    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:01:05.157946 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:05.157897    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq" podStartSLOduration=2.157880146 podStartE2EDuration="2.157880146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:05.156214473 +0000 UTC m=+1286.016039153" watchObservedRunningTime="2026-09-17 19:01:05.157880146 +0000 UTC m=+1286.017704793"
Sep 17 19:01:10.996726 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:10.996698    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:01:36.142760 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:01:36.142730    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:02:03.486592 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:03.486507    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"]
Sep 17 19:02:10.996233 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:10.996193    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c" podUID="139dcb7e-43da-4b63-b1ec-494029f24309" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:02:18.506849 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:18.506794    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c" podUID="139dcb7e-43da-4b63-b1ec-494029f24309" containerName="main" containerID="cri-o://bb379ac3afc366b815c2999b45363409c55d1d1b9aabf215d6c8b254a379e3d5" gracePeriod=45
Sep 17 19:02:20.996003 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:20.995963    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c" podUID="139dcb7e-43da-4b63-b1ec-494029f24309" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:02:30.997033 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:30.996984    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c" podUID="139dcb7e-43da-4b63-b1ec-494029f24309" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:02:30.997480 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:30.997103    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:02:40.996299 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:40.996263    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c" podUID="139dcb7e-43da-4b63-b1ec-494029f24309" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:02:48.704926 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:48.704866    2586 generic.go:356] "Generic (PLEG): container finished" podID="139dcb7e-43da-4b63-b1ec-494029f24309" containerID="bb379ac3afc366b815c2999b45363409c55d1d1b9aabf215d6c8b254a379e3d5" exitCode=0
Sep 17 19:02:48.705309 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:48.704876    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c" event={"ID":"139dcb7e-43da-4b63-b1ec-494029f24309","Type":"ContainerDied","Data":"bb379ac3afc366b815c2999b45363409c55d1d1b9aabf215d6c8b254a379e3d5"}
Sep 17 19:02:48.761359 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:48.761336    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:02:48.807625 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:48.807549    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xltrr\" (UniqueName: \"kubernetes.io/projected/139dcb7e-43da-4b63-b1ec-494029f24309-kube-api-access-xltrr\") pod \"139dcb7e-43da-4b63-b1ec-494029f24309\" (UID: \"139dcb7e-43da-4b63-b1ec-494029f24309\") "
Sep 17 19:02:48.807625 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:48.807601    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139dcb7e-43da-4b63-b1ec-494029f24309-tls-certs\") pod \"139dcb7e-43da-4b63-b1ec-494029f24309\" (UID: \"139dcb7e-43da-4b63-b1ec-494029f24309\") "
Sep 17 19:02:48.810108 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:48.810079    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139dcb7e-43da-4b63-b1ec-494029f24309-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "139dcb7e-43da-4b63-b1ec-494029f24309" (UID: "139dcb7e-43da-4b63-b1ec-494029f24309"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:48.810225 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:48.810112    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139dcb7e-43da-4b63-b1ec-494029f24309-kube-api-access-xltrr" (OuterVolumeSpecName: "kube-api-access-xltrr") pod "139dcb7e-43da-4b63-b1ec-494029f24309" (UID: "139dcb7e-43da-4b63-b1ec-494029f24309"). InnerVolumeSpecName "kube-api-access-xltrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:48.909233 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:48.909198    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xltrr\" (UniqueName: \"kubernetes.io/projected/139dcb7e-43da-4b63-b1ec-494029f24309-kube-api-access-xltrr\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:02:48.909233 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:48.909229    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139dcb7e-43da-4b63-b1ec-494029f24309-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:02:49.717487 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:49.717393    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c" event={"ID":"139dcb7e-43da-4b63-b1ec-494029f24309","Type":"ContainerDied","Data":"3ba0bd506209da9653efbc1d395bc71ab858582e85aa1f2c34d3c15f259cdf8b"}
Sep 17 19:02:49.717487 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:49.717404    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"
Sep 17 19:02:49.718017 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:49.717482    2586 scope.go:117] "RemoveContainer" containerID="bb379ac3afc366b815c2999b45363409c55d1d1b9aabf215d6c8b254a379e3d5"
Sep 17 19:02:49.740957 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:49.740930    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"]
Sep 17 19:02:49.745581 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:49.745557    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b6979d5c"]
Sep 17 19:02:51.459709 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:02:51.459677    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="139dcb7e-43da-4b63-b1ec-494029f24309" path="/var/lib/kubelet/pods/139dcb7e-43da-4b63-b1ec-494029f24309/volumes"
Sep 17 19:03:04.910198 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.910153    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"]
Sep 17 19:03:04.910742 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.910722    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="139dcb7e-43da-4b63-b1ec-494029f24309" containerName="main"
Sep 17 19:03:04.910852 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.910746    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="139dcb7e-43da-4b63-b1ec-494029f24309" containerName="main"
Sep 17 19:03:04.910908 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.910875    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="139dcb7e-43da-4b63-b1ec-494029f24309" containerName="main"
Sep 17 19:03:04.914025 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.914001    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:03:04.918450 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.918428    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 19:03:04.918672 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.918655    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 19:03:04.919966 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.919941    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-dbndq\""
Sep 17 19:03:04.920077 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.919950    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 19:03:04.920573 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.920554    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-4nf2q\""
Sep 17 19:03:04.930400 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:04.930374    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"]
Sep 17 19:03:05.049213 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.049178    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6\" (UID: \"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:03:05.049400 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.049226    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mgcd\" (UniqueName: \"kubernetes.io/projected/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-kube-api-access-9mgcd\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6\" (UID: \"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:03:05.149787 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.149748    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6\" (UID: \"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:03:05.150008 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.149798    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgcd\" (UniqueName: \"kubernetes.io/projected/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-kube-api-access-9mgcd\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6\" (UID: \"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:03:05.152460 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.152435    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6\" (UID: \"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:03:05.162412 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.162348    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgcd\" (UniqueName: \"kubernetes.io/projected/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-kube-api-access-9mgcd\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6\" (UID: \"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:03:05.225346 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.225305    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:03:05.506297 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.506266    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"]
Sep 17 19:03:05.507363 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:03:05.507335    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c529ca7_0bc9_4efc_9fe5_d76f138f1e97.slice/crio-11e311da145e717b212d03bdc3564028035c8a2d20eed8742bc00e10dfaac6d9 WatchSource:0}: Error finding container 11e311da145e717b212d03bdc3564028035c8a2d20eed8742bc00e10dfaac6d9: Status 404 returned error can't find the container with id 11e311da145e717b212d03bdc3564028035c8a2d20eed8742bc00e10dfaac6d9
Sep 17 19:03:05.812294 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.812211    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" event={"ID":"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97","Type":"ContainerStarted","Data":"c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4"}
Sep 17 19:03:05.812294 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.812266    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" event={"ID":"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97","Type":"ContainerStarted","Data":"11e311da145e717b212d03bdc3564028035c8a2d20eed8742bc00e10dfaac6d9"}
Sep 17 19:03:05.812478 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.812300    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:03:05.861072 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:05.861018    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" podStartSLOduration=1.861001532 podStartE2EDuration="1.861001532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:05.85777331 +0000 UTC m=+1406.717597978" watchObservedRunningTime="2026-09-17 19:03:05.861001532 +0000 UTC m=+1406.720826179"
Sep 17 19:03:36.819671 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:36.819595    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:03:42.257272 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:42.257238    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"]
Sep 17 19:03:46.261892 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:46.261861    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"]
Sep 17 19:03:46.819828 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:46.819795    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" podUID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:56.142433 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:56.142396    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq" podUID="9eed5df7-e9bd-47a8-b68a-9920bc484335" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:56.819609 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:56.819562    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" podUID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:57.281443 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:03:57.281352    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" podUID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" containerName="main" containerID="cri-o://c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4" gracePeriod=45
Sep 17 19:04:01.282300 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:01.282228    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq" podUID="9eed5df7-e9bd-47a8-b68a-9920bc484335" containerName="main" containerID="cri-o://b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb" gracePeriod=45
Sep 17 19:04:06.142862 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:06.142803    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq" podUID="9eed5df7-e9bd-47a8-b68a-9920bc484335" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:04:06.819723 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:06.819684    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" podUID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:04:06.819969 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:06.819816    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:04:16.142958 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:16.142915    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq" podUID="9eed5df7-e9bd-47a8-b68a-9920bc484335" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:04:16.143431 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:16.143038    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:04:16.819017 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:16.818985    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" podUID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:04:26.142952 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:26.142913    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq" podUID="9eed5df7-e9bd-47a8-b68a-9920bc484335" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:04:26.820065 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:26.820017    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" podUID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:04:27.541909 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:27.541821    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:04:27.573816 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:27.573783    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-tls-certs\") pod \"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97\" (UID: \"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97\") "
Sep 17 19:04:27.574017 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:27.573828    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mgcd\" (UniqueName: \"kubernetes.io/projected/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-kube-api-access-9mgcd\") pod \"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97\" (UID: \"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97\") "
Sep 17 19:04:27.576255 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:27.576221    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" (UID: "7c529ca7-0bc9-4efc-9fe5-d76f138f1e97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:27.576255 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:27.576235    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-kube-api-access-9mgcd" (OuterVolumeSpecName: "kube-api-access-9mgcd") pod "7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" (UID: "7c529ca7-0bc9-4efc-9fe5-d76f138f1e97"). InnerVolumeSpecName "kube-api-access-9mgcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:27.674687 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:27.674650    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:04:27.674687 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:27.674681    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mgcd\" (UniqueName: \"kubernetes.io/projected/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97-kube-api-access-9mgcd\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:04:28.264418 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:28.264382    2586 generic.go:356] "Generic (PLEG): container finished" podID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" containerID="c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4" exitCode=0
Sep 17 19:04:28.264613 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:28.264423    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"
Sep 17 19:04:28.264613 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:28.264472    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" event={"ID":"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97","Type":"ContainerDied","Data":"c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4"}
Sep 17 19:04:28.264613 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:28.264518    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6" event={"ID":"7c529ca7-0bc9-4efc-9fe5-d76f138f1e97","Type":"ContainerDied","Data":"11e311da145e717b212d03bdc3564028035c8a2d20eed8742bc00e10dfaac6d9"}
Sep 17 19:04:28.264613 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:28.264541    2586 scope.go:117] "RemoveContainer" containerID="c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4"
Sep 17 19:04:28.274453 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:28.274436    2586 scope.go:117] "RemoveContainer" containerID="c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4"
Sep 17 19:04:28.274728 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:04:28.274710    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4\": container with ID starting with c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4 not found: ID does not exist" containerID="c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4"
Sep 17 19:04:28.274775 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:28.274734    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4"} err="failed to get container status \"c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4\": rpc error: code = NotFound desc = could not find container \"c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4\": container with ID starting with c2caa88b5322844109eaa4a441dd50bfeeba39720159cae138f6be03e732abd4 not found: ID does not exist"
Sep 17 19:04:28.293799 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:28.293770    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"]
Sep 17 19:04:28.297216 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:28.297191    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wmwj6"]
Sep 17 19:04:29.459055 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:29.459019    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" path="/var/lib/kubelet/pods/7c529ca7-0bc9-4efc-9fe5-d76f138f1e97/volumes"
Sep 17 19:04:31.537391 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:31.537363    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:04:31.611284 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:31.611250    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eed5df7-e9bd-47a8-b68a-9920bc484335-tls-certs\") pod \"9eed5df7-e9bd-47a8-b68a-9920bc484335\" (UID: \"9eed5df7-e9bd-47a8-b68a-9920bc484335\") "
Sep 17 19:04:31.611284 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:31.611286    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9598x\" (UniqueName: \"kubernetes.io/projected/9eed5df7-e9bd-47a8-b68a-9920bc484335-kube-api-access-9598x\") pod \"9eed5df7-e9bd-47a8-b68a-9920bc484335\" (UID: \"9eed5df7-e9bd-47a8-b68a-9920bc484335\") "
Sep 17 19:04:31.613713 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:31.613677    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eed5df7-e9bd-47a8-b68a-9920bc484335-kube-api-access-9598x" (OuterVolumeSpecName: "kube-api-access-9598x") pod "9eed5df7-e9bd-47a8-b68a-9920bc484335" (UID: "9eed5df7-e9bd-47a8-b68a-9920bc484335"). InnerVolumeSpecName "kube-api-access-9598x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:31.613713 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:31.613684    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eed5df7-e9bd-47a8-b68a-9920bc484335-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9eed5df7-e9bd-47a8-b68a-9920bc484335" (UID: "9eed5df7-e9bd-47a8-b68a-9920bc484335"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:31.712694 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:31.712661    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eed5df7-e9bd-47a8-b68a-9920bc484335-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:04:31.712694 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:31.712690    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9598x\" (UniqueName: \"kubernetes.io/projected/9eed5df7-e9bd-47a8-b68a-9920bc484335-kube-api-access-9598x\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:04:32.288259 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:32.288224    2586 generic.go:356] "Generic (PLEG): container finished" podID="9eed5df7-e9bd-47a8-b68a-9920bc484335" containerID="b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb" exitCode=0
Sep 17 19:04:32.288442 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:32.288269    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"
Sep 17 19:04:32.288442 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:32.288304    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq" event={"ID":"9eed5df7-e9bd-47a8-b68a-9920bc484335","Type":"ContainerDied","Data":"b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb"}
Sep 17 19:04:32.288442 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:32.288353    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq" event={"ID":"9eed5df7-e9bd-47a8-b68a-9920bc484335","Type":"ContainerDied","Data":"b2cc1030b51ec85471653308c2c145a68c19af3dac0cb585a5ce95f02856557e"}
Sep 17 19:04:32.288442 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:32.288375    2586 scope.go:117] "RemoveContainer" containerID="b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb"
Sep 17 19:04:32.297814 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:32.297794    2586 scope.go:117] "RemoveContainer" containerID="b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb"
Sep 17 19:04:32.298171 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:04:32.298149    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb\": container with ID starting with b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb not found: ID does not exist" containerID="b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb"
Sep 17 19:04:32.298249 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:32.298185    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb"} err="failed to get container status \"b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb\": rpc error: code = NotFound desc = could not find container \"b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb\": container with ID starting with b18fb726ee2b0e76efd915748af17bef51d3ec3d872e0c17c8501e9d3a8f9dfb not found: ID does not exist"
Sep 17 19:04:32.313212 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:32.313183    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"]
Sep 17 19:04:32.319503 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:32.319479    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch28rq"]
Sep 17 19:04:33.458687 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:33.458652    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eed5df7-e9bd-47a8-b68a-9920bc484335" path="/var/lib/kubelet/pods/9eed5df7-e9bd-47a8-b68a-9920bc484335/volumes"
Sep 17 19:04:41.468612 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:41.468582    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:04:41.475882 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:41.475859    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:04:57.859408 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.859303    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"]
Sep 17 19:04:57.859818 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.859701    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eed5df7-e9bd-47a8-b68a-9920bc484335" containerName="main"
Sep 17 19:04:57.859818 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.859713    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eed5df7-e9bd-47a8-b68a-9920bc484335" containerName="main"
Sep 17 19:04:57.859818 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.859731    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" containerName="main"
Sep 17 19:04:57.859818 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.859737    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" containerName="main"
Sep 17 19:04:57.859818 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.859814    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c529ca7-0bc9-4efc-9fe5-d76f138f1e97" containerName="main"
Sep 17 19:04:57.860103 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.859871    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eed5df7-e9bd-47a8-b68a-9920bc484335" containerName="main"
Sep 17 19:04:57.864580 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.864558    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:04:57.869104 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.869075    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-gzg4z\""
Sep 17 19:04:57.870437 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.870416    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-pbvt9\""
Sep 17 19:04:57.870541 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.870435    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 19:04:57.870541 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.870488    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 19:04:57.870541 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.870520    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 19:04:57.879104 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.879079    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"]
Sep 17 19:04:57.947136 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.947093    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvkw\" (UniqueName: \"kubernetes.io/projected/1c40971b-086e-4e47-8e08-75e8c2dca67a-kube-api-access-thvkw\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd\" (UID: \"1c40971b-086e-4e47-8e08-75e8c2dca67a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:04:57.947315 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:57.947152    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c40971b-086e-4e47-8e08-75e8c2dca67a-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd\" (UID: \"1c40971b-086e-4e47-8e08-75e8c2dca67a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:04:58.047986 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:58.047936    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvkw\" (UniqueName: \"kubernetes.io/projected/1c40971b-086e-4e47-8e08-75e8c2dca67a-kube-api-access-thvkw\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd\" (UID: \"1c40971b-086e-4e47-8e08-75e8c2dca67a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:04:58.047986 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:58.047996    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c40971b-086e-4e47-8e08-75e8c2dca67a-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd\" (UID: \"1c40971b-086e-4e47-8e08-75e8c2dca67a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:04:58.050890 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:58.050864    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c40971b-086e-4e47-8e08-75e8c2dca67a-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd\" (UID: \"1c40971b-086e-4e47-8e08-75e8c2dca67a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:04:58.060560 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:58.060531    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvkw\" (UniqueName: \"kubernetes.io/projected/1c40971b-086e-4e47-8e08-75e8c2dca67a-kube-api-access-thvkw\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd\" (UID: \"1c40971b-086e-4e47-8e08-75e8c2dca67a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:04:58.175626 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:58.175539    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:04:58.464847 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:58.464811    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"]
Sep 17 19:04:58.467147 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:04:58.467119    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c40971b_086e_4e47_8e08_75e8c2dca67a.slice/crio-058df3daabe9e33809b7349d9bbb0b7bb975813e6f74501157d7a69ab59eeea6 WatchSource:0}: Error finding container 058df3daabe9e33809b7349d9bbb0b7bb975813e6f74501157d7a69ab59eeea6: Status 404 returned error can't find the container with id 058df3daabe9e33809b7349d9bbb0b7bb975813e6f74501157d7a69ab59eeea6
Sep 17 19:04:59.445533 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:59.445493    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd" event={"ID":"1c40971b-086e-4e47-8e08-75e8c2dca67a","Type":"ContainerStarted","Data":"bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73"}
Sep 17 19:04:59.445998 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:59.445586    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd" event={"ID":"1c40971b-086e-4e47-8e08-75e8c2dca67a","Type":"ContainerStarted","Data":"058df3daabe9e33809b7349d9bbb0b7bb975813e6f74501157d7a69ab59eeea6"}
Sep 17 19:04:59.445998 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:04:59.445618    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:05:18.275331 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.275267    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd" podStartSLOduration=21.275247161 podStartE2EDuration="21.275247161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:59.509880782 +0000 UTC m=+1520.369705449" watchObservedRunningTime="2026-09-17 19:05:18.275247161 +0000 UTC m=+1539.135071808"
Sep 17 19:05:18.277288 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.277269    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"]
Sep 17 19:05:18.280749 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.280729    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:05:18.284335 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.284306    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-w9tfb\""
Sep 17 19:05:18.284335 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.284323    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 19:05:18.284507 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.284319    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-bk9g9\""
Sep 17 19:05:18.284507 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.284346    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 19:05:18.284507 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.284342    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 19:05:18.295095 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.295069    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"]
Sep 17 19:05:18.322792 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.322762    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc5z4\" (UniqueName: \"kubernetes.io/projected/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-kube-api-access-bc5z4\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6\" (UID: \"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:05:18.322973 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.322828    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6\" (UID: \"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:05:18.424161 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.424119    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc5z4\" (UniqueName: \"kubernetes.io/projected/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-kube-api-access-bc5z4\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6\" (UID: \"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:05:18.424342 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.424222    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6\" (UID: \"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:05:18.426935 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.426901    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6\" (UID: \"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:05:18.432740 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.432715    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc5z4\" (UniqueName: \"kubernetes.io/projected/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-kube-api-access-bc5z4\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6\" (UID: \"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:05:18.593186 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.593151    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:05:18.873017 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:18.872987    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"]
Sep 17 19:05:18.874470 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:05:18.874449    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62773c71_ec2c_4b5d_9b5c_5c01b7e711bb.slice/crio-a7907091218e0b83bcf208129bf73d150d3377d9761ba2688520bbbad92c92aa WatchSource:0}: Error finding container a7907091218e0b83bcf208129bf73d150d3377d9761ba2688520bbbad92c92aa: Status 404 returned error can't find the container with id a7907091218e0b83bcf208129bf73d150d3377d9761ba2688520bbbad92c92aa
Sep 17 19:05:19.560894 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:19.560824    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" event={"ID":"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb","Type":"ContainerStarted","Data":"e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055"}
Sep 17 19:05:19.561322 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:19.560909    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" event={"ID":"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb","Type":"ContainerStarted","Data":"a7907091218e0b83bcf208129bf73d150d3377d9761ba2688520bbbad92c92aa"}
Sep 17 19:05:19.561322 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:19.560942    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:05:19.581038 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:19.580983    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" podStartSLOduration=1.5809647930000001 podStartE2EDuration="1.580964793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:19.578421969 +0000 UTC m=+1540.438246618" watchObservedRunningTime="2026-09-17 19:05:19.580964793 +0000 UTC m=+1540.440789440"
Sep 17 19:05:30.456803 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:30.456774    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:05:50.569381 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:50.569347    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:05:52.289791 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:05:52.289752    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"]
Sep 17 19:06:00.456195 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:00.456153    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd" podUID="1c40971b-086e-4e47-8e08-75e8c2dca67a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:06:07.320797 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:07.320729    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd" podUID="1c40971b-086e-4e47-8e08-75e8c2dca67a" containerName="main" containerID="cri-o://bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73" gracePeriod=45
Sep 17 19:06:10.456068 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:10.456027    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd" podUID="1c40971b-086e-4e47-8e08-75e8c2dca67a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:06:20.456875 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:20.456758    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd" podUID="1c40971b-086e-4e47-8e08-75e8c2dca67a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:06:20.457337 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:20.456901    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:06:30.456661 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:30.456623    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd" podUID="1c40971b-086e-4e47-8e08-75e8c2dca67a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:06:37.575022 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.574995    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:06:37.711105 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.710995    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thvkw\" (UniqueName: \"kubernetes.io/projected/1c40971b-086e-4e47-8e08-75e8c2dca67a-kube-api-access-thvkw\") pod \"1c40971b-086e-4e47-8e08-75e8c2dca67a\" (UID: \"1c40971b-086e-4e47-8e08-75e8c2dca67a\") "
Sep 17 19:06:37.711105 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.711043    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c40971b-086e-4e47-8e08-75e8c2dca67a-tls-certs\") pod \"1c40971b-086e-4e47-8e08-75e8c2dca67a\" (UID: \"1c40971b-086e-4e47-8e08-75e8c2dca67a\") "
Sep 17 19:06:37.713561 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.713522    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c40971b-086e-4e47-8e08-75e8c2dca67a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c40971b-086e-4e47-8e08-75e8c2dca67a" (UID: "1c40971b-086e-4e47-8e08-75e8c2dca67a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:06:37.713700 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.713606    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c40971b-086e-4e47-8e08-75e8c2dca67a-kube-api-access-thvkw" (OuterVolumeSpecName: "kube-api-access-thvkw") pod "1c40971b-086e-4e47-8e08-75e8c2dca67a" (UID: "1c40971b-086e-4e47-8e08-75e8c2dca67a"). InnerVolumeSpecName "kube-api-access-thvkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:06:37.812070 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.812033    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thvkw\" (UniqueName: \"kubernetes.io/projected/1c40971b-086e-4e47-8e08-75e8c2dca67a-kube-api-access-thvkw\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:06:37.812070 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.812062    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c40971b-086e-4e47-8e08-75e8c2dca67a-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:06:37.997887 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.997772    2586 generic.go:356] "Generic (PLEG): container finished" podID="1c40971b-086e-4e47-8e08-75e8c2dca67a" containerID="bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73" exitCode=0
Sep 17 19:06:37.997887 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.997818    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"
Sep 17 19:06:37.997887 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.997859    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd" event={"ID":"1c40971b-086e-4e47-8e08-75e8c2dca67a","Type":"ContainerDied","Data":"bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73"}
Sep 17 19:06:37.998248 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.997921    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd" event={"ID":"1c40971b-086e-4e47-8e08-75e8c2dca67a","Type":"ContainerDied","Data":"058df3daabe9e33809b7349d9bbb0b7bb975813e6f74501157d7a69ab59eeea6"}
Sep 17 19:06:37.998248 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:37.997946    2586 scope.go:117] "RemoveContainer" containerID="bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73"
Sep 17 19:06:38.008368 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:38.008347    2586 scope.go:117] "RemoveContainer" containerID="bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73"
Sep 17 19:06:38.008717 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:06:38.008693    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73\": container with ID starting with bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73 not found: ID does not exist" containerID="bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73"
Sep 17 19:06:38.008775 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:38.008725    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73"} err="failed to get container status \"bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73\": rpc error: code = NotFound desc = could not find container \"bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73\": container with ID starting with bdcc420f14503d364069ac435e5a3d6d908c343ea83a6d89931a08e35666fc73 not found: ID does not exist"
Sep 17 19:06:38.024165 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:38.024141    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"]
Sep 17 19:06:38.028792 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:38.028769    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dm9lvd"]
Sep 17 19:06:39.458620 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:06:39.458587    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c40971b-086e-4e47-8e08-75e8c2dca67a" path="/var/lib/kubelet/pods/1c40971b-086e-4e47-8e08-75e8c2dca67a/volumes"
Sep 17 19:07:56.145911 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:07:56.145806    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"]
Sep 17 19:08:00.569340 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:00.569305    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" podUID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:08:10.569120 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:10.569083    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" podUID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:08:11.166970 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:11.166897    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" podUID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" containerName="main" containerID="cri-o://e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055" gracePeriod=45
Sep 17 19:08:20.569162 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:20.569121    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" podUID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:08:20.569637 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:20.569250    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:08:30.569699 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:30.569657    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" podUID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:08:40.569347 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:40.569308    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" podUID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:08:41.430361 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.430303    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:08:41.494433 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.494395    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-tls-certs\") pod \"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb\" (UID: \"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb\") "
Sep 17 19:08:41.494630 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.494467    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc5z4\" (UniqueName: \"kubernetes.io/projected/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-kube-api-access-bc5z4\") pod \"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb\" (UID: \"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb\") "
Sep 17 19:08:41.496969 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.496941    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" (UID: "62773c71-ec2c-4b5d-9b5c-5c01b7e711bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:41.496969 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.496959    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-kube-api-access-bc5z4" (OuterVolumeSpecName: "kube-api-access-bc5z4") pod "62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" (UID: "62773c71-ec2c-4b5d-9b5c-5c01b7e711bb"). InnerVolumeSpecName "kube-api-access-bc5z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:41.595883 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.595810    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc5z4\" (UniqueName: \"kubernetes.io/projected/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-kube-api-access-bc5z4\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:08:41.595883 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.595875    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:08:41.677387 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.677349    2586 generic.go:356] "Generic (PLEG): container finished" podID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" containerID="e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055" exitCode=0
Sep 17 19:08:41.677561 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.677400    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"
Sep 17 19:08:41.677561 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.677418    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" event={"ID":"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb","Type":"ContainerDied","Data":"e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055"}
Sep 17 19:08:41.677561 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.677469    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6" event={"ID":"62773c71-ec2c-4b5d-9b5c-5c01b7e711bb","Type":"ContainerDied","Data":"a7907091218e0b83bcf208129bf73d150d3377d9761ba2688520bbbad92c92aa"}
Sep 17 19:08:41.677561 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.677501    2586 scope.go:117] "RemoveContainer" containerID="e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055"
Sep 17 19:08:41.686735 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.686718    2586 scope.go:117] "RemoveContainer" containerID="e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055"
Sep 17 19:08:41.687037 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:08:41.687015    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055\": container with ID starting with e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055 not found: ID does not exist" containerID="e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055"
Sep 17 19:08:41.687117 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.687051    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055"} err="failed to get container status \"e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055\": rpc error: code = NotFound desc = could not find container \"e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055\": container with ID starting with e0c6cea083f596c4198a2a23187963ded93b1027ba7c5b334009b18962ff3055 not found: ID does not exist"
Sep 17 19:08:41.700732 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.700700    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"]
Sep 17 19:08:41.703295 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:41.703264    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-tbch6"]
Sep 17 19:08:43.459030 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:43.458997    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" path="/var/lib/kubelet/pods/62773c71-ec2c-4b5d-9b5c-5c01b7e711bb/volumes"
Sep 17 19:08:49.277601 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.277513    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"]
Sep 17 19:08:49.278028 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.277991    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c40971b-086e-4e47-8e08-75e8c2dca67a" containerName="main"
Sep 17 19:08:49.278028 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.278009    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c40971b-086e-4e47-8e08-75e8c2dca67a" containerName="main"
Sep 17 19:08:49.278139 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.278036    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" containerName="main"
Sep 17 19:08:49.278139 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.278046    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" containerName="main"
Sep 17 19:08:49.278139 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.278133    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c40971b-086e-4e47-8e08-75e8c2dca67a" containerName="main"
Sep 17 19:08:49.278364 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.278152    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="62773c71-ec2c-4b5d-9b5c-5c01b7e711bb" containerName="main"
Sep 17 19:08:49.282775 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.282757    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:08:49.286358 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.286337    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-cm2wn\""
Sep 17 19:08:49.286491 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.286367    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-x5xqb\""
Sep 17 19:08:49.286491 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.286416    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 19:08:49.286491 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.286373    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 19:08:49.286491 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.286452    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 19:08:49.294332 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.294308    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"]
Sep 17 19:08:49.364214 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.364172    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clfqd\" (UniqueName: \"kubernetes.io/projected/91e42588-743d-4926-96ea-31daa8d2062c-kube-api-access-clfqd\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx\" (UID: \"91e42588-743d-4926-96ea-31daa8d2062c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:08:49.364420 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.364278    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e42588-743d-4926-96ea-31daa8d2062c-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx\" (UID: \"91e42588-743d-4926-96ea-31daa8d2062c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:08:49.465145 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.465107    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e42588-743d-4926-96ea-31daa8d2062c-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx\" (UID: \"91e42588-743d-4926-96ea-31daa8d2062c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:08:49.465311 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.465198    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clfqd\" (UniqueName: \"kubernetes.io/projected/91e42588-743d-4926-96ea-31daa8d2062c-kube-api-access-clfqd\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx\" (UID: \"91e42588-743d-4926-96ea-31daa8d2062c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:08:49.467898 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.467873    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e42588-743d-4926-96ea-31daa8d2062c-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx\" (UID: \"91e42588-743d-4926-96ea-31daa8d2062c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:08:49.474210 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.474181    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clfqd\" (UniqueName: \"kubernetes.io/projected/91e42588-743d-4926-96ea-31daa8d2062c-kube-api-access-clfqd\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx\" (UID: \"91e42588-743d-4926-96ea-31daa8d2062c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:08:49.594656 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.594609    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:08:49.872468 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:49.872381    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"]
Sep 17 19:08:49.875571 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:08:49.875545    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e42588_743d_4926_96ea_31daa8d2062c.slice/crio-19cc408121f84f1ff234089ccc2ef164de05a7820ac1e97de361808596cbc4b0 WatchSource:0}: Error finding container 19cc408121f84f1ff234089ccc2ef164de05a7820ac1e97de361808596cbc4b0: Status 404 returned error can't find the container with id 19cc408121f84f1ff234089ccc2ef164de05a7820ac1e97de361808596cbc4b0
Sep 17 19:08:50.727883 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:50.727828    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx" event={"ID":"91e42588-743d-4926-96ea-31daa8d2062c","Type":"ContainerStarted","Data":"8660b9c69a0dc4efcb2aaf27b4ced1b14dfecf4e9bae86b0129bcbf6f09a9c90"}
Sep 17 19:08:50.728290 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:50.727900    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx" event={"ID":"91e42588-743d-4926-96ea-31daa8d2062c","Type":"ContainerStarted","Data":"19cc408121f84f1ff234089ccc2ef164de05a7820ac1e97de361808596cbc4b0"}
Sep 17 19:08:50.728290 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:50.727987    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:08:50.748644 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:08:50.748591    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx" podStartSLOduration=1.748574397 podStartE2EDuration="1.748574397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:50.747756455 +0000 UTC m=+1751.607581120" watchObservedRunningTime="2026-09-17 19:08:50.748574397 +0000 UTC m=+1751.608399045"
Sep 17 19:09:21.735031 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:21.734949    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:09:36.790950 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.790895    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"]
Sep 17 19:09:36.794446 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.794429    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.797264 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.797239    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 19:09:36.806070 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.806045    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"]
Sep 17 19:09:36.889157 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.889120    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.889339 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.889162    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.889339 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.889208    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.889339 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.889233    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.889339 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.889280    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnkq\" (UniqueName: \"kubernetes.io/projected/5e4897c3-eab8-475e-93f2-7377b53c71e8-kube-api-access-5qnkq\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.889339 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.889299    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4897c3-eab8-475e-93f2-7377b53c71e8-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.990669 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.990634    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.990669 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.990675    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.990922 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.990696    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.990922 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.990712    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.990922 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.990745    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnkq\" (UniqueName: \"kubernetes.io/projected/5e4897c3-eab8-475e-93f2-7377b53c71e8-kube-api-access-5qnkq\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.990922 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.990761    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4897c3-eab8-475e-93f2-7377b53c71e8-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.991156 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.991132    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.991204 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.991189    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.991242 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.991206    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.993174 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.993158    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.993535 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.993518    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4897c3-eab8-475e-93f2-7377b53c71e8-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:36.999381 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:36.999361    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnkq\" (UniqueName: \"kubernetes.io/projected/5e4897c3-eab8-475e-93f2-7377b53c71e8-kube-api-access-5qnkq\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:37.105668 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:37.105632    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:37.387055 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:37.387023    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"]
Sep 17 19:09:37.387967 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:09:37.387943    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4897c3_eab8_475e_93f2_7377b53c71e8.slice/crio-aafaa3fa41411f44851e70236a452f09c6b7ca393ac8e8500f1b18c439e4249c WatchSource:0}: Error finding container aafaa3fa41411f44851e70236a452f09c6b7ca393ac8e8500f1b18c439e4249c: Status 404 returned error can't find the container with id aafaa3fa41411f44851e70236a452f09c6b7ca393ac8e8500f1b18c439e4249c
Sep 17 19:09:37.988929 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:37.988886    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52" event={"ID":"5e4897c3-eab8-475e-93f2-7377b53c71e8","Type":"ContainerStarted","Data":"d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325"}
Sep 17 19:09:37.989368 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:37.988951    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52" event={"ID":"5e4897c3-eab8-475e-93f2-7377b53c71e8","Type":"ContainerStarted","Data":"aafaa3fa41411f44851e70236a452f09c6b7ca393ac8e8500f1b18c439e4249c"}
Sep 17 19:09:38.010460 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:38.010400    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52" podStartSLOduration=2.010383555 podStartE2EDuration="2.010383555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:38.007803074 +0000 UTC m=+1798.867627735" watchObservedRunningTime="2026-09-17 19:09:38.010383555 +0000 UTC m=+1798.870208203"
Sep 17 19:09:38.106012 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:38.105975    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:41.506407 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:41.506376    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:09:41.514357 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:41.514336    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:09:47.106151 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:47.106112    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:47.118678 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:47.118651    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:09:48.058080 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:09:48.058049    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:10:45.414945 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.414910    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"]
Sep 17 19:10:45.424947 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:10:45.424902    2586 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:10:45.424947 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:10:45.424947 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:10:45.424947 ip-10-0-129-204 kubenswrapper[2586]:  > containerID="d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325" cmd=["/bin/sleep","15"]
Sep 17 19:10:45.425185 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.424961    2586 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52" message="" err=<
Sep 17 19:10:45.425185 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:10:45.425185 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:10:45.425185 ip-10-0-129-204 kubenswrapper[2586]:  >
Sep 17 19:10:45.425185 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:10:45.424976    2586 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:10:45.425185 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:10:45.425185 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:10:45.425185 ip-10-0-129-204 kubenswrapper[2586]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52" podUID="5e4897c3-eab8-475e-93f2-7377b53c71e8" containerName="main" containerID="cri-o://d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325"
Sep 17 19:10:45.425185 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.425030    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52" podUID="5e4897c3-eab8-475e-93f2-7377b53c71e8" containerName="main" containerID="cri-o://d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325" gracePeriod=60
Sep 17 19:10:45.782874 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.782845    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:10:45.910801 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.910771    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4897c3-eab8-475e-93f2-7377b53c71e8-tls-certs\") pod \"5e4897c3-eab8-475e-93f2-7377b53c71e8\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") "
Sep 17 19:10:45.911014 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.910822    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-model-cache\") pod \"5e4897c3-eab8-475e-93f2-7377b53c71e8\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") "
Sep 17 19:10:45.911014 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.910897    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qnkq\" (UniqueName: \"kubernetes.io/projected/5e4897c3-eab8-475e-93f2-7377b53c71e8-kube-api-access-5qnkq\") pod \"5e4897c3-eab8-475e-93f2-7377b53c71e8\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") "
Sep 17 19:10:45.911014 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.910944    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-home\") pod \"5e4897c3-eab8-475e-93f2-7377b53c71e8\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") "
Sep 17 19:10:45.911014 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.910973    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-dshm\") pod \"5e4897c3-eab8-475e-93f2-7377b53c71e8\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") "
Sep 17 19:10:45.911014 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.911001    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-tmp-dir\") pod \"5e4897c3-eab8-475e-93f2-7377b53c71e8\" (UID: \"5e4897c3-eab8-475e-93f2-7377b53c71e8\") "
Sep 17 19:10:45.911284 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.911197    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e4897c3-eab8-475e-93f2-7377b53c71e8" (UID: "5e4897c3-eab8-475e-93f2-7377b53c71e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:45.911284 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.911217    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-home" (OuterVolumeSpecName: "home") pod "5e4897c3-eab8-475e-93f2-7377b53c71e8" (UID: "5e4897c3-eab8-475e-93f2-7377b53c71e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:45.911284 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.911263    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:10:45.911408 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.911346    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e4897c3-eab8-475e-93f2-7377b53c71e8" (UID: "5e4897c3-eab8-475e-93f2-7377b53c71e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:45.913394 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.913363    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e4897c3-eab8-475e-93f2-7377b53c71e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e4897c3-eab8-475e-93f2-7377b53c71e8" (UID: "5e4897c3-eab8-475e-93f2-7377b53c71e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:10:45.913394 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.913370    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4897c3-eab8-475e-93f2-7377b53c71e8-kube-api-access-5qnkq" (OuterVolumeSpecName: "kube-api-access-5qnkq") pod "5e4897c3-eab8-475e-93f2-7377b53c71e8" (UID: "5e4897c3-eab8-475e-93f2-7377b53c71e8"). InnerVolumeSpecName "kube-api-access-5qnkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:10:45.913394 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:45.913368    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-dshm" (OuterVolumeSpecName: "dshm") pod "5e4897c3-eab8-475e-93f2-7377b53c71e8" (UID: "5e4897c3-eab8-475e-93f2-7377b53c71e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:46.012508 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.012421    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:10:46.012508 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.012452    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:10:46.012508 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.012461    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4897c3-eab8-475e-93f2-7377b53c71e8-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:10:46.012508 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.012470    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4897c3-eab8-475e-93f2-7377b53c71e8-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:10:46.012508 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.012481    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnkq\" (UniqueName: \"kubernetes.io/projected/5e4897c3-eab8-475e-93f2-7377b53c71e8-kube-api-access-5qnkq\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:10:46.382100 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.382063    2586 generic.go:356] "Generic (PLEG): container finished" podID="5e4897c3-eab8-475e-93f2-7377b53c71e8" containerID="d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325" exitCode=0
Sep 17 19:10:46.382285 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.382107    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"
Sep 17 19:10:46.382285 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.382158    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52" event={"ID":"5e4897c3-eab8-475e-93f2-7377b53c71e8","Type":"ContainerDied","Data":"d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325"}
Sep 17 19:10:46.382285 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.382209    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52" event={"ID":"5e4897c3-eab8-475e-93f2-7377b53c71e8","Type":"ContainerDied","Data":"aafaa3fa41411f44851e70236a452f09c6b7ca393ac8e8500f1b18c439e4249c"}
Sep 17 19:10:46.382285 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.382231    2586 scope.go:117] "RemoveContainer" containerID="d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325"
Sep 17 19:10:46.392701 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.392686    2586 scope.go:117] "RemoveContainer" containerID="d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325"
Sep 17 19:10:46.393021 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:10:46.393001    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325\": container with ID starting with d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325 not found: ID does not exist" containerID="d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325"
Sep 17 19:10:46.393075 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.393029    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325"} err="failed to get container status \"d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325\": rpc error: code = NotFound desc = could not find container \"d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325\": container with ID starting with d403acb1a11d1a6a3adf4541f2be626b50b18d913c6cdecb3f7ff0c593462325 not found: ID does not exist"
Sep 17 19:10:46.407630 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.407606    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"]
Sep 17 19:10:46.411374 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:46.411353    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xdq52"]
Sep 17 19:10:47.459609 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:10:47.459570    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e4897c3-eab8-475e-93f2-7377b53c71e8" path="/var/lib/kubelet/pods/5e4897c3-eab8-475e-93f2-7377b53c71e8/volumes"
Sep 17 19:11:33.629307 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:11:33.629272    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"]
Sep 17 19:11:41.734935 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:11:41.734896    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx" podUID="91e42588-743d-4926-96ea-31daa8d2062c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:11:48.650341 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:11:48.650301    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx" podUID="91e42588-743d-4926-96ea-31daa8d2062c" containerName="main" containerID="cri-o://8660b9c69a0dc4efcb2aaf27b4ced1b14dfecf4e9bae86b0129bcbf6f09a9c90" gracePeriod=45
Sep 17 19:11:51.735517 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:11:51.735480    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx" podUID="91e42588-743d-4926-96ea-31daa8d2062c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:12:01.735340 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:01.735297    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx" podUID="91e42588-743d-4926-96ea-31daa8d2062c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:12:01.735752 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:01.735406    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:12:11.735366 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:11.735321    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx" podUID="91e42588-743d-4926-96ea-31daa8d2062c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:12:18.900036 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:18.899953    2586 generic.go:356] "Generic (PLEG): container finished" podID="91e42588-743d-4926-96ea-31daa8d2062c" containerID="8660b9c69a0dc4efcb2aaf27b4ced1b14dfecf4e9bae86b0129bcbf6f09a9c90" exitCode=0
Sep 17 19:12:18.900036 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:18.900023    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx" event={"ID":"91e42588-743d-4926-96ea-31daa8d2062c","Type":"ContainerDied","Data":"8660b9c69a0dc4efcb2aaf27b4ced1b14dfecf4e9bae86b0129bcbf6f09a9c90"}
Sep 17 19:12:18.900462 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:18.900073    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx" event={"ID":"91e42588-743d-4926-96ea-31daa8d2062c","Type":"ContainerDied","Data":"19cc408121f84f1ff234089ccc2ef164de05a7820ac1e97de361808596cbc4b0"}
Sep 17 19:12:18.900462 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:18.900089    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19cc408121f84f1ff234089ccc2ef164de05a7820ac1e97de361808596cbc4b0"
Sep 17 19:12:18.900462 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:18.900199    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:12:19.023178 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:19.023143    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clfqd\" (UniqueName: \"kubernetes.io/projected/91e42588-743d-4926-96ea-31daa8d2062c-kube-api-access-clfqd\") pod \"91e42588-743d-4926-96ea-31daa8d2062c\" (UID: \"91e42588-743d-4926-96ea-31daa8d2062c\") "
Sep 17 19:12:19.023379 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:19.023235    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e42588-743d-4926-96ea-31daa8d2062c-tls-certs\") pod \"91e42588-743d-4926-96ea-31daa8d2062c\" (UID: \"91e42588-743d-4926-96ea-31daa8d2062c\") "
Sep 17 19:12:19.025483 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:19.025448    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e42588-743d-4926-96ea-31daa8d2062c-kube-api-access-clfqd" (OuterVolumeSpecName: "kube-api-access-clfqd") pod "91e42588-743d-4926-96ea-31daa8d2062c" (UID: "91e42588-743d-4926-96ea-31daa8d2062c"). InnerVolumeSpecName "kube-api-access-clfqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:12:19.025593 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:19.025498    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e42588-743d-4926-96ea-31daa8d2062c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91e42588-743d-4926-96ea-31daa8d2062c" (UID: "91e42588-743d-4926-96ea-31daa8d2062c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:12:19.124413 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:19.124373    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clfqd\" (UniqueName: \"kubernetes.io/projected/91e42588-743d-4926-96ea-31daa8d2062c-kube-api-access-clfqd\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:12:19.124413 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:19.124410    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e42588-743d-4926-96ea-31daa8d2062c-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:12:19.905310 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:19.905276    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"
Sep 17 19:12:19.922768 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:19.922736    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"]
Sep 17 19:12:19.926675 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:19.926645    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetdkcx"]
Sep 17 19:12:21.458935 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:21.458895    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e42588-743d-4926-96ea-31daa8d2062c" path="/var/lib/kubelet/pods/91e42588-743d-4926-96ea-31daa8d2062c/volumes"
Sep 17 19:12:29.822740 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.822708    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"]
Sep 17 19:12:29.823163 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.823146    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e4897c3-eab8-475e-93f2-7377b53c71e8" containerName="main"
Sep 17 19:12:29.823208 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.823165    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e4897c3-eab8-475e-93f2-7377b53c71e8" containerName="main"
Sep 17 19:12:29.823208 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.823174    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e42588-743d-4926-96ea-31daa8d2062c" containerName="main"
Sep 17 19:12:29.823208 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.823179    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e42588-743d-4926-96ea-31daa8d2062c" containerName="main"
Sep 17 19:12:29.823303 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.823256    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e42588-743d-4926-96ea-31daa8d2062c" containerName="main"
Sep 17 19:12:29.823303 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.823263    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e4897c3-eab8-475e-93f2-7377b53c71e8" containerName="main"
Sep 17 19:12:29.826552 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.826533    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:12:29.830343 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.830318    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 19:12:29.830472 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.830345    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-x7zbj\""
Sep 17 19:12:29.830472 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.830356    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 19:12:29.830472 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.830358    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-xjfsk\""
Sep 17 19:12:29.830472 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.830328    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 19:12:29.838542 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.838515    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"]
Sep 17 19:12:29.923566 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.923530    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a78bd0-559f-45ad-9cca-a2c3576f865f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq\" (UID: \"f3a78bd0-559f-45ad-9cca-a2c3576f865f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:12:29.923764 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:29.923587    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl85n\" (UniqueName: \"kubernetes.io/projected/f3a78bd0-559f-45ad-9cca-a2c3576f865f-kube-api-access-dl85n\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq\" (UID: \"f3a78bd0-559f-45ad-9cca-a2c3576f865f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:12:30.024188 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:30.024142    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl85n\" (UniqueName: \"kubernetes.io/projected/f3a78bd0-559f-45ad-9cca-a2c3576f865f-kube-api-access-dl85n\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq\" (UID: \"f3a78bd0-559f-45ad-9cca-a2c3576f865f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:12:30.024375 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:30.024265    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a78bd0-559f-45ad-9cca-a2c3576f865f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq\" (UID: \"f3a78bd0-559f-45ad-9cca-a2c3576f865f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:12:30.027064 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:30.027039    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a78bd0-559f-45ad-9cca-a2c3576f865f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq\" (UID: \"f3a78bd0-559f-45ad-9cca-a2c3576f865f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:12:30.034989 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:30.034969    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl85n\" (UniqueName: \"kubernetes.io/projected/f3a78bd0-559f-45ad-9cca-a2c3576f865f-kube-api-access-dl85n\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq\" (UID: \"f3a78bd0-559f-45ad-9cca-a2c3576f865f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:12:30.138976 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:30.138883    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:12:30.418355 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:30.418327    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"]
Sep 17 19:12:30.420531 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:12:30.420504    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a78bd0_559f_45ad_9cca_a2c3576f865f.slice/crio-d2e13b04be1171526613f2b4041c8cadc7b006f377322509cd4f37b6fc6c53ea WatchSource:0}: Error finding container d2e13b04be1171526613f2b4041c8cadc7b006f377322509cd4f37b6fc6c53ea: Status 404 returned error can't find the container with id d2e13b04be1171526613f2b4041c8cadc7b006f377322509cd4f37b6fc6c53ea
Sep 17 19:12:30.968856 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:30.968799    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" event={"ID":"f3a78bd0-559f-45ad-9cca-a2c3576f865f","Type":"ContainerStarted","Data":"d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2"}
Sep 17 19:12:30.969268 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:30.968863    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" event={"ID":"f3a78bd0-559f-45ad-9cca-a2c3576f865f","Type":"ContainerStarted","Data":"d2e13b04be1171526613f2b4041c8cadc7b006f377322509cd4f37b6fc6c53ea"}
Sep 17 19:12:30.969268 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:30.968884    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:12:30.988576 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:12:30.988525    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" podStartSLOduration=1.988511159 podStartE2EDuration="1.988511159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:30.986333828 +0000 UTC m=+1971.846158498" watchObservedRunningTime="2026-09-17 19:12:30.988511159 +0000 UTC m=+1971.848335806"
Sep 17 19:13:01.976180 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:01.976151    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:13:10.809737 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.809699    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"]
Sep 17 19:13:10.813798 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.813772    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:13:10.816574 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.816552    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-r98qm\""
Sep 17 19:13:10.817534 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.817512    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 19:13:10.830619 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.830582    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"]
Sep 17 19:13:10.883397 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.883332    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hnd\" (UniqueName: \"kubernetes.io/projected/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-kube-api-access-p7hnd\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x\" (UID: \"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:13:10.883397 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.883401    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x\" (UID: \"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:13:10.984001 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.983961    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hnd\" (UniqueName: \"kubernetes.io/projected/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-kube-api-access-p7hnd\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x\" (UID: \"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:13:10.984001 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.984002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x\" (UID: \"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:13:10.986694 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.986673    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x\" (UID: \"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:13:10.993045 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:10.993021    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hnd\" (UniqueName: \"kubernetes.io/projected/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-kube-api-access-p7hnd\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x\" (UID: \"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:13:11.124326 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:11.124286    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:13:11.400581 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:11.400555    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"]
Sep 17 19:13:11.402364 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:13:11.402339    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c6f8f6_74c5_4d98_a1e4_bc2810a1f9f8.slice/crio-eef49ffba8d7bb0186e8c985e47d77b0b7adaa71c465856d5337afc0ecaf020f WatchSource:0}: Error finding container eef49ffba8d7bb0186e8c985e47d77b0b7adaa71c465856d5337afc0ecaf020f: Status 404 returned error can't find the container with id eef49ffba8d7bb0186e8c985e47d77b0b7adaa71c465856d5337afc0ecaf020f
Sep 17 19:13:12.197750 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:12.197714    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" event={"ID":"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8","Type":"ContainerStarted","Data":"395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f"}
Sep 17 19:13:12.197750 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:12.197770    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" event={"ID":"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8","Type":"ContainerStarted","Data":"eef49ffba8d7bb0186e8c985e47d77b0b7adaa71c465856d5337afc0ecaf020f"}
Sep 17 19:13:12.198219 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:12.197789    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:13:12.217519 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:12.217467    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" podStartSLOduration=2.217450286 podStartE2EDuration="2.217450286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:12.214941303 +0000 UTC m=+2013.074765970" watchObservedRunningTime="2026-09-17 19:13:12.217450286 +0000 UTC m=+2013.077274961"
Sep 17 19:13:43.205525 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:43.205493    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:13:59.802350 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:13:59.802261    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"]
Sep 17 19:14:03.205998 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:03.205959    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" podUID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:14:13.205184 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:13.205142    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" podUID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:14:14.830059 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:14.830020    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" podUID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" containerName="main" containerID="cri-o://395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f" gracePeriod=45
Sep 17 19:14:23.205653 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:23.205603    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" podUID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:14:23.206100 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:23.205721    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:14:33.205151 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:33.205105    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" podUID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:14:41.542824 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:41.542793    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:14:41.552997 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:41.552978    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:14:43.205470 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:43.205432    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" podUID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:14:45.087900 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.087821    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:14:45.185920 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.185888    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-tls-certs\") pod \"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8\" (UID: \"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8\") "
Sep 17 19:14:45.186121 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.185937    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7hnd\" (UniqueName: \"kubernetes.io/projected/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-kube-api-access-p7hnd\") pod \"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8\" (UID: \"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8\") "
Sep 17 19:14:45.188301 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.188274    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" (UID: "09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:14:45.188408 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.188302    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-kube-api-access-p7hnd" (OuterVolumeSpecName: "kube-api-access-p7hnd") pod "09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" (UID: "09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8"). InnerVolumeSpecName "kube-api-access-p7hnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:45.286721 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.286678    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:14:45.286721 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.286714    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7hnd\" (UniqueName: \"kubernetes.io/projected/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8-kube-api-access-p7hnd\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:14:45.707537 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.707502    2586 generic.go:356] "Generic (PLEG): container finished" podID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" containerID="395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f" exitCode=0
Sep 17 19:14:45.707714 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.707557    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"
Sep 17 19:14:45.707714 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.707567    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" event={"ID":"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8","Type":"ContainerDied","Data":"395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f"}
Sep 17 19:14:45.707714 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.707612    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x" event={"ID":"09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8","Type":"ContainerDied","Data":"eef49ffba8d7bb0186e8c985e47d77b0b7adaa71c465856d5337afc0ecaf020f"}
Sep 17 19:14:45.707714 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.707633    2586 scope.go:117] "RemoveContainer" containerID="395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f"
Sep 17 19:14:45.717431 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.717410    2586 scope.go:117] "RemoveContainer" containerID="395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f"
Sep 17 19:14:45.717707 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:14:45.717689    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f\": container with ID starting with 395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f not found: ID does not exist" containerID="395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f"
Sep 17 19:14:45.717760 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.717716    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f"} err="failed to get container status \"395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f\": rpc error: code = NotFound desc = could not find container \"395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f\": container with ID starting with 395cab74e30abf80f3a8eb59bc276d8830478b0025d2e68a6b566ed9462b144f not found: ID does not exist"
Sep 17 19:14:45.725821 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.725792    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"]
Sep 17 19:14:45.731893 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:45.731866    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jx4x"]
Sep 17 19:14:47.459390 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:47.459355    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" path="/var/lib/kubelet/pods/09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8/volumes"
Sep 17 19:14:49.259904 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:49.259866    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"]
Sep 17 19:14:51.976308 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:14:51.976272    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" podUID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:15:01.977044 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:01.976998    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" podUID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:15:04.280752 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:04.280684    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" podUID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" containerName="main" containerID="cri-o://d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2" gracePeriod=45
Sep 17 19:15:11.976296 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:11.976255    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" podUID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:15:11.976795 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:11.976359    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:15:21.976307 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:21.976219    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" podUID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:15:31.975968 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:31.975928    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" podUID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:15:34.523190 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.523160    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:15:34.704937 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.704896    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl85n\" (UniqueName: \"kubernetes.io/projected/f3a78bd0-559f-45ad-9cca-a2c3576f865f-kube-api-access-dl85n\") pod \"f3a78bd0-559f-45ad-9cca-a2c3576f865f\" (UID: \"f3a78bd0-559f-45ad-9cca-a2c3576f865f\") "
Sep 17 19:15:34.705125 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.705013    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a78bd0-559f-45ad-9cca-a2c3576f865f-tls-certs\") pod \"f3a78bd0-559f-45ad-9cca-a2c3576f865f\" (UID: \"f3a78bd0-559f-45ad-9cca-a2c3576f865f\") "
Sep 17 19:15:34.707214 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.707178    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a78bd0-559f-45ad-9cca-a2c3576f865f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3a78bd0-559f-45ad-9cca-a2c3576f865f" (UID: "f3a78bd0-559f-45ad-9cca-a2c3576f865f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:15:34.707334 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.707221    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a78bd0-559f-45ad-9cca-a2c3576f865f-kube-api-access-dl85n" (OuterVolumeSpecName: "kube-api-access-dl85n") pod "f3a78bd0-559f-45ad-9cca-a2c3576f865f" (UID: "f3a78bd0-559f-45ad-9cca-a2c3576f865f"). InnerVolumeSpecName "kube-api-access-dl85n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:15:34.805915 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.805878    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a78bd0-559f-45ad-9cca-a2c3576f865f-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:15:34.805915 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.805904    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl85n\" (UniqueName: \"kubernetes.io/projected/f3a78bd0-559f-45ad-9cca-a2c3576f865f-kube-api-access-dl85n\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:15:34.986556 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.986470    2586 generic.go:356] "Generic (PLEG): container finished" podID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" containerID="d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2" exitCode=0
Sep 17 19:15:34.986556 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.986514    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"
Sep 17 19:15:34.986766 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.986515    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" event={"ID":"f3a78bd0-559f-45ad-9cca-a2c3576f865f","Type":"ContainerDied","Data":"d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2"}
Sep 17 19:15:34.986766 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.986652    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq" event={"ID":"f3a78bd0-559f-45ad-9cca-a2c3576f865f","Type":"ContainerDied","Data":"d2e13b04be1171526613f2b4041c8cadc7b006f377322509cd4f37b6fc6c53ea"}
Sep 17 19:15:34.986766 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.986681    2586 scope.go:117] "RemoveContainer" containerID="d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2"
Sep 17 19:15:34.996587 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.996572    2586 scope.go:117] "RemoveContainer" containerID="d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2"
Sep 17 19:15:34.996876 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:15:34.996852    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2\": container with ID starting with d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2 not found: ID does not exist" containerID="d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2"
Sep 17 19:15:34.996948 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:34.996887    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2"} err="failed to get container status \"d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2\": rpc error: code = NotFound desc = could not find container \"d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2\": container with ID starting with d73d6f44602117a15a13a4fdcd7c4e7abefb880d64d984d537d412c1a9eed7c2 not found: ID does not exist"
Sep 17 19:15:35.008816 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:35.008782    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"]
Sep 17 19:15:35.012339 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:35.012313    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche2krnq"]
Sep 17 19:15:35.458298 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:35.458263    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" path="/var/lib/kubelet/pods/f3a78bd0-559f-45ad-9cca-a2c3576f865f/volumes"
Sep 17 19:15:41.667221 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:41.667188    2586 scope.go:117] "RemoveContainer" containerID="8660b9c69a0dc4efcb2aaf27b4ced1b14dfecf4e9bae86b0129bcbf6f09a9c90"
Sep 17 19:15:51.768402 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.768366    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"]
Sep 17 19:15:51.769850 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.769798    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" containerName="main"
Sep 17 19:15:51.769850 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.769854    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" containerName="main"
Sep 17 19:15:51.770051 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.769933    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" containerName="main"
Sep 17 19:15:51.770051 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.769945    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" containerName="main"
Sep 17 19:15:51.771958 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.770150    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c6f8f6-74c5-4d98-a1e4-bc2810a1f9f8" containerName="main"
Sep 17 19:15:51.771958 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.770178    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a78bd0-559f-45ad-9cca-a2c3576f865f" containerName="main"
Sep 17 19:15:51.775638 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.775616    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:15:51.783600 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.783579    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-nn965\""
Sep 17 19:15:51.784733 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.784714    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 19:15:51.784889 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.784802    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 19:15:51.784889 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.784853    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-str4z\""
Sep 17 19:15:51.785068 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.784956    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 19:15:51.787444 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.787422    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"]
Sep 17 19:15:51.840404 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.840367    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd75e8a-f958-4deb-887f-10761628e112-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g\" (UID: \"ddd75e8a-f958-4deb-887f-10761628e112\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:15:51.840404 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.840410    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7n4l\" (UniqueName: \"kubernetes.io/projected/ddd75e8a-f958-4deb-887f-10761628e112-kube-api-access-l7n4l\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g\" (UID: \"ddd75e8a-f958-4deb-887f-10761628e112\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:15:51.941507 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.941460    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7n4l\" (UniqueName: \"kubernetes.io/projected/ddd75e8a-f958-4deb-887f-10761628e112-kube-api-access-l7n4l\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g\" (UID: \"ddd75e8a-f958-4deb-887f-10761628e112\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:15:51.941732 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.941594    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd75e8a-f958-4deb-887f-10761628e112-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g\" (UID: \"ddd75e8a-f958-4deb-887f-10761628e112\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:15:51.944483 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.944454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd75e8a-f958-4deb-887f-10761628e112-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g\" (UID: \"ddd75e8a-f958-4deb-887f-10761628e112\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:15:51.956303 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:51.956277    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7n4l\" (UniqueName: \"kubernetes.io/projected/ddd75e8a-f958-4deb-887f-10761628e112-kube-api-access-l7n4l\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g\" (UID: \"ddd75e8a-f958-4deb-887f-10761628e112\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:15:52.087692 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:52.087646    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:15:52.372121 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:52.372092    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"]
Sep 17 19:15:52.374418 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:15:52.374389    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd75e8a_f958_4deb_887f_10761628e112.slice/crio-875847c8c748a5415a3eeba3d81be759ba8234914f345537a9e7ea2e615315e5 WatchSource:0}: Error finding container 875847c8c748a5415a3eeba3d81be759ba8234914f345537a9e7ea2e615315e5: Status 404 returned error can't find the container with id 875847c8c748a5415a3eeba3d81be759ba8234914f345537a9e7ea2e615315e5
Sep 17 19:15:53.084730 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:53.084687    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g" event={"ID":"ddd75e8a-f958-4deb-887f-10761628e112","Type":"ContainerStarted","Data":"91ca17156b98490955dab4bea45c9e989540fa603d57586356b62124bf5a0b2d"}
Sep 17 19:15:53.085215 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:53.084751    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g" event={"ID":"ddd75e8a-f958-4deb-887f-10761628e112","Type":"ContainerStarted","Data":"875847c8c748a5415a3eeba3d81be759ba8234914f345537a9e7ea2e615315e5"}
Sep 17 19:15:53.085215 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:53.084785    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:15:53.110479 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:15:53.110425    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g" podStartSLOduration=2.110409856 podStartE2EDuration="2.110409856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:53.108447345 +0000 UTC m=+2173.968271994" watchObservedRunningTime="2026-09-17 19:15:53.110409856 +0000 UTC m=+2173.970234535"
Sep 17 19:16:24.093344 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:16:24.093311    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:17:47.760511 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:17:47.760471    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"]
Sep 17 19:17:54.093517 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:17:54.093479    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g" podUID="ddd75e8a-f958-4deb-887f-10761628e112" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:18:02.780264 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:02.780198    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g" podUID="ddd75e8a-f958-4deb-887f-10761628e112" containerName="main" containerID="cri-o://91ca17156b98490955dab4bea45c9e989540fa603d57586356b62124bf5a0b2d" gracePeriod=45
Sep 17 19:18:04.093182 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:04.093137    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g" podUID="ddd75e8a-f958-4deb-887f-10761628e112" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:18:14.092699 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:14.092658    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g" podUID="ddd75e8a-f958-4deb-887f-10761628e112" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:18:14.093140 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:14.092770    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:18:24.093398 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:24.093313    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g" podUID="ddd75e8a-f958-4deb-887f-10761628e112" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:18:32.965037 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:32.965000    2586 generic.go:356] "Generic (PLEG): container finished" podID="ddd75e8a-f958-4deb-887f-10761628e112" containerID="91ca17156b98490955dab4bea45c9e989540fa603d57586356b62124bf5a0b2d" exitCode=0
Sep 17 19:18:32.965386 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:32.965067    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g" event={"ID":"ddd75e8a-f958-4deb-887f-10761628e112","Type":"ContainerDied","Data":"91ca17156b98490955dab4bea45c9e989540fa603d57586356b62124bf5a0b2d"}
Sep 17 19:18:33.037162 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.037132    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:18:33.139907 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.139873    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7n4l\" (UniqueName: \"kubernetes.io/projected/ddd75e8a-f958-4deb-887f-10761628e112-kube-api-access-l7n4l\") pod \"ddd75e8a-f958-4deb-887f-10761628e112\" (UID: \"ddd75e8a-f958-4deb-887f-10761628e112\") "
Sep 17 19:18:33.139907 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.139911    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd75e8a-f958-4deb-887f-10761628e112-tls-certs\") pod \"ddd75e8a-f958-4deb-887f-10761628e112\" (UID: \"ddd75e8a-f958-4deb-887f-10761628e112\") "
Sep 17 19:18:33.142311 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.142286    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd75e8a-f958-4deb-887f-10761628e112-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddd75e8a-f958-4deb-887f-10761628e112" (UID: "ddd75e8a-f958-4deb-887f-10761628e112"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:18:33.142410 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.142376    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd75e8a-f958-4deb-887f-10761628e112-kube-api-access-l7n4l" (OuterVolumeSpecName: "kube-api-access-l7n4l") pod "ddd75e8a-f958-4deb-887f-10761628e112" (UID: "ddd75e8a-f958-4deb-887f-10761628e112"). InnerVolumeSpecName "kube-api-access-l7n4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:18:33.240723 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.240685    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7n4l\" (UniqueName: \"kubernetes.io/projected/ddd75e8a-f958-4deb-887f-10761628e112-kube-api-access-l7n4l\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:18:33.240723 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.240715    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd75e8a-f958-4deb-887f-10761628e112-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:18:33.973201 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.973166    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g" event={"ID":"ddd75e8a-f958-4deb-887f-10761628e112","Type":"ContainerDied","Data":"875847c8c748a5415a3eeba3d81be759ba8234914f345537a9e7ea2e615315e5"}
Sep 17 19:18:33.973628 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.973225    2586 scope.go:117] "RemoveContainer" containerID="91ca17156b98490955dab4bea45c9e989540fa603d57586356b62124bf5a0b2d"
Sep 17 19:18:33.973628 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.973179    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"
Sep 17 19:18:33.994620 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:33.994594    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"]
Sep 17 19:18:34.000419 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:34.000396    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schehms6g"]
Sep 17 19:18:35.458958 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:35.458927    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd75e8a-f958-4deb-887f-10761628e112" path="/var/lib/kubelet/pods/ddd75e8a-f958-4deb-887f-10761628e112/volumes"
Sep 17 19:18:58.430705 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.430669    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"]
Sep 17 19:18:58.431356 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.431082    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd75e8a-f958-4deb-887f-10761628e112" containerName="main"
Sep 17 19:18:58.431356 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.431094    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd75e8a-f958-4deb-887f-10761628e112" containerName="main"
Sep 17 19:18:58.431356 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.431158    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd75e8a-f958-4deb-887f-10761628e112" containerName="main"
Sep 17 19:18:58.435341 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.435321    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:18:58.439211 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.439190    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 19:18:58.439763 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.439635    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-b5scb\""
Sep 17 19:18:58.439763 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.439693    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 19:18:58.439763 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.439693    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 19:18:58.440049 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.439693    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-ztkff\""
Sep 17 19:18:58.446043 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.446022    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"]
Sep 17 19:18:58.563316 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.563285    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1f14b24-8892-4fe4-b65e-9d8148532002-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v\" (UID: \"d1f14b24-8892-4fe4-b65e-9d8148532002\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:18:58.563316 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.563318    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5qxz\" (UniqueName: \"kubernetes.io/projected/d1f14b24-8892-4fe4-b65e-9d8148532002-kube-api-access-g5qxz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v\" (UID: \"d1f14b24-8892-4fe4-b65e-9d8148532002\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:18:58.664736 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.664699    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1f14b24-8892-4fe4-b65e-9d8148532002-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v\" (UID: \"d1f14b24-8892-4fe4-b65e-9d8148532002\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:18:58.664736 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.664736    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5qxz\" (UniqueName: \"kubernetes.io/projected/d1f14b24-8892-4fe4-b65e-9d8148532002-kube-api-access-g5qxz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v\" (UID: \"d1f14b24-8892-4fe4-b65e-9d8148532002\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:18:58.667498 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.667466    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1f14b24-8892-4fe4-b65e-9d8148532002-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v\" (UID: \"d1f14b24-8892-4fe4-b65e-9d8148532002\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:18:58.678022 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.677983    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5qxz\" (UniqueName: \"kubernetes.io/projected/d1f14b24-8892-4fe4-b65e-9d8148532002-kube-api-access-g5qxz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v\" (UID: \"d1f14b24-8892-4fe4-b65e-9d8148532002\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:18:58.746718 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:58.746620    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:18:59.029064 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:59.029038    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"]
Sep 17 19:18:59.031245 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:18:59.031219    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f14b24_8892_4fe4_b65e_9d8148532002.slice/crio-68a028faee4ea0b523a6096f5cff1ccb7aa94dea00bee0550bfedf4fe16ed1b8 WatchSource:0}: Error finding container 68a028faee4ea0b523a6096f5cff1ccb7aa94dea00bee0550bfedf4fe16ed1b8: Status 404 returned error can't find the container with id 68a028faee4ea0b523a6096f5cff1ccb7aa94dea00bee0550bfedf4fe16ed1b8
Sep 17 19:18:59.126603 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:59.126486    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" event={"ID":"d1f14b24-8892-4fe4-b65e-9d8148532002","Type":"ContainerStarted","Data":"9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0"}
Sep 17 19:18:59.126603 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:59.126567    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" event={"ID":"d1f14b24-8892-4fe4-b65e-9d8148532002","Type":"ContainerStarted","Data":"68a028faee4ea0b523a6096f5cff1ccb7aa94dea00bee0550bfedf4fe16ed1b8"}
Sep 17 19:18:59.126603 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:59.126600    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:18:59.160899 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:18:59.160815    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" podStartSLOduration=1.160797806 podStartE2EDuration="1.160797806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:18:59.160648919 +0000 UTC m=+2360.020473587" watchObservedRunningTime="2026-09-17 19:18:59.160797806 +0000 UTC m=+2360.020622453"
Sep 17 19:19:30.134587 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:19:30.134555    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:19:41.587709 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:19:41.587685    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:19:41.595703 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:19:41.595681    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:20:32.165044 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.165009    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"]
Sep 17 19:20:32.168550 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.168533    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:20:32.172717 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.172548    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 19:20:32.172717 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.172577    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-pqw24\""
Sep 17 19:20:32.172717 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.172585    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-h4tdj\""
Sep 17 19:20:32.172717 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.172586    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 19:20:32.172717 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.172557    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 19:20:32.176615 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.176575    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"]
Sep 17 19:20:32.273714 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.273673    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7hrb\" (UniqueName: \"kubernetes.io/projected/121cf523-18bf-4ac8-99b6-0973d3768d44-kube-api-access-d7hrb\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9\" (UID: \"121cf523-18bf-4ac8-99b6-0973d3768d44\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:20:32.273929 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.273722    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121cf523-18bf-4ac8-99b6-0973d3768d44-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9\" (UID: \"121cf523-18bf-4ac8-99b6-0973d3768d44\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:20:32.374554 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.374514    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121cf523-18bf-4ac8-99b6-0973d3768d44-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9\" (UID: \"121cf523-18bf-4ac8-99b6-0973d3768d44\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:20:32.374727 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.374595    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7hrb\" (UniqueName: \"kubernetes.io/projected/121cf523-18bf-4ac8-99b6-0973d3768d44-kube-api-access-d7hrb\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9\" (UID: \"121cf523-18bf-4ac8-99b6-0973d3768d44\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:20:32.377290 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.377269    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121cf523-18bf-4ac8-99b6-0973d3768d44-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9\" (UID: \"121cf523-18bf-4ac8-99b6-0973d3768d44\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:20:32.383990 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.383956    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7hrb\" (UniqueName: \"kubernetes.io/projected/121cf523-18bf-4ac8-99b6-0973d3768d44-kube-api-access-d7hrb\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9\" (UID: \"121cf523-18bf-4ac8-99b6-0973d3768d44\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:20:32.480813 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.480726    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:20:32.756704 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:32.756676    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"]
Sep 17 19:20:32.758254 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:20:32.758225    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121cf523_18bf_4ac8_99b6_0973d3768d44.slice/crio-e4cdb3d9df67fb6e93725a23df03c45aee58b995c9d72951f4aead80e35a8bf9 WatchSource:0}: Error finding container e4cdb3d9df67fb6e93725a23df03c45aee58b995c9d72951f4aead80e35a8bf9: Status 404 returned error can't find the container with id e4cdb3d9df67fb6e93725a23df03c45aee58b995c9d72951f4aead80e35a8bf9
Sep 17 19:20:33.661178 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:33.661134    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9" event={"ID":"121cf523-18bf-4ac8-99b6-0973d3768d44","Type":"ContainerStarted","Data":"30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76"}
Sep 17 19:20:33.661686 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:33.661237    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9" event={"ID":"121cf523-18bf-4ac8-99b6-0973d3768d44","Type":"ContainerStarted","Data":"e4cdb3d9df67fb6e93725a23df03c45aee58b995c9d72951f4aead80e35a8bf9"}
Sep 17 19:20:33.661939 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:33.661917    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:20:33.682922 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:33.682869    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9" podStartSLOduration=1.6828547600000001 podStartE2EDuration="1.68285476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:33.68075662 +0000 UTC m=+2454.540581286" watchObservedRunningTime="2026-09-17 19:20:33.68285476 +0000 UTC m=+2454.542679402"
Sep 17 19:20:46.236306 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:46.236266    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"]
Sep 17 19:20:50.134636 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:20:50.134590    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" podUID="d1f14b24-8892-4fe4-b65e-9d8148532002" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:00.135307 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:00.135259    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" podUID="d1f14b24-8892-4fe4-b65e-9d8148532002" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:01.256875 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:01.256799    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" podUID="d1f14b24-8892-4fe4-b65e-9d8148532002" containerName="main" containerID="cri-o://9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0" gracePeriod=45
Sep 17 19:21:04.668188 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:04.668160    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:21:10.135181 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:10.135136    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" podUID="d1f14b24-8892-4fe4-b65e-9d8148532002" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:10.135604 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:10.135258    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:21:20.135169 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:20.135122    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" podUID="d1f14b24-8892-4fe4-b65e-9d8148532002" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:30.134589 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:30.134547    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" podUID="d1f14b24-8892-4fe4-b65e-9d8148532002" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:31.512569 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.512544    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:21:31.585444 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.585408    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5qxz\" (UniqueName: \"kubernetes.io/projected/d1f14b24-8892-4fe4-b65e-9d8148532002-kube-api-access-g5qxz\") pod \"d1f14b24-8892-4fe4-b65e-9d8148532002\" (UID: \"d1f14b24-8892-4fe4-b65e-9d8148532002\") "
Sep 17 19:21:31.585635 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.585489    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1f14b24-8892-4fe4-b65e-9d8148532002-tls-certs\") pod \"d1f14b24-8892-4fe4-b65e-9d8148532002\" (UID: \"d1f14b24-8892-4fe4-b65e-9d8148532002\") "
Sep 17 19:21:31.587929 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.587892    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f14b24-8892-4fe4-b65e-9d8148532002-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1f14b24-8892-4fe4-b65e-9d8148532002" (UID: "d1f14b24-8892-4fe4-b65e-9d8148532002"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:21:31.588053 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.587942    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f14b24-8892-4fe4-b65e-9d8148532002-kube-api-access-g5qxz" (OuterVolumeSpecName: "kube-api-access-g5qxz") pod "d1f14b24-8892-4fe4-b65e-9d8148532002" (UID: "d1f14b24-8892-4fe4-b65e-9d8148532002"). InnerVolumeSpecName "kube-api-access-g5qxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:21:31.686668 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.686632    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1f14b24-8892-4fe4-b65e-9d8148532002-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:21:31.686668 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.686662    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5qxz\" (UniqueName: \"kubernetes.io/projected/d1f14b24-8892-4fe4-b65e-9d8148532002-kube-api-access-g5qxz\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:21:31.983428 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.983324    2586 generic.go:356] "Generic (PLEG): container finished" podID="d1f14b24-8892-4fe4-b65e-9d8148532002" containerID="9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0" exitCode=0
Sep 17 19:21:31.983428 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.983373    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" event={"ID":"d1f14b24-8892-4fe4-b65e-9d8148532002","Type":"ContainerDied","Data":"9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0"}
Sep 17 19:21:31.983428 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.983375    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"
Sep 17 19:21:31.983428 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.983417    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v" event={"ID":"d1f14b24-8892-4fe4-b65e-9d8148532002","Type":"ContainerDied","Data":"68a028faee4ea0b523a6096f5cff1ccb7aa94dea00bee0550bfedf4fe16ed1b8"}
Sep 17 19:21:31.983772 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.983438    2586 scope.go:117] "RemoveContainer" containerID="9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0"
Sep 17 19:21:31.996140 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.996107    2586 scope.go:117] "RemoveContainer" containerID="9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0"
Sep 17 19:21:31.997345 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:21:31.997314    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0\": container with ID starting with 9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0 not found: ID does not exist" containerID="9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0"
Sep 17 19:21:31.997444 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:31.997356    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0"} err="failed to get container status \"9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0\": rpc error: code = NotFound desc = could not find container \"9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0\": container with ID starting with 9f7dce09131fd5a1b6014457f176877dced72c591db400d4eafcad7d8825b9e0 not found: ID does not exist"
Sep 17 19:21:32.011625 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:32.011595    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"]
Sep 17 19:21:32.014984 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:32.014959    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq7s9v"]
Sep 17 19:21:33.459077 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:33.459042    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1f14b24-8892-4fe4-b65e-9d8148532002" path="/var/lib/kubelet/pods/d1f14b24-8892-4fe4-b65e-9d8148532002/volumes"
Sep 17 19:21:44.078668 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.078372    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"]
Sep 17 19:21:44.079246 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.078958    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f14b24-8892-4fe4-b65e-9d8148532002" containerName="main"
Sep 17 19:21:44.079246 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.078975    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f14b24-8892-4fe4-b65e-9d8148532002" containerName="main"
Sep 17 19:21:44.079246 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.079090    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f14b24-8892-4fe4-b65e-9d8148532002" containerName="main"
Sep 17 19:21:44.083817 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.083792    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:21:44.088041 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.088011    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 19:21:44.088215 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.088187    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 19:21:44.088499 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.088475    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-fflm7\""
Sep 17 19:21:44.088598 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.088578    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-wpgxx\""
Sep 17 19:21:44.088663 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.088639    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 19:21:44.090731 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.090710    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"]
Sep 17 19:21:44.197437 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.197401    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkkk\" (UniqueName: \"kubernetes.io/projected/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-kube-api-access-xzkkk\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v\" (UID: \"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:21:44.197625 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.197485    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v\" (UID: \"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:21:44.298359 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.298323    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v\" (UID: \"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:21:44.298537 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.298377    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkkk\" (UniqueName: \"kubernetes.io/projected/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-kube-api-access-xzkkk\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v\" (UID: \"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:21:44.301107 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.301077    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v\" (UID: \"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:21:44.307726 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.307693    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkkk\" (UniqueName: \"kubernetes.io/projected/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-kube-api-access-xzkkk\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v\" (UID: \"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:21:44.396863 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.396750    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:21:44.674923 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:44.674889    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"]
Sep 17 19:21:44.682128 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:21:44.682097    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d1d617_bd19_4f42_aa77_ac04dc7e5dce.slice/crio-da85dbb4c991180c35dcb8c1d0e0dc8eb47d251d6284a195d374f7abab879dd9 WatchSource:0}: Error finding container da85dbb4c991180c35dcb8c1d0e0dc8eb47d251d6284a195d374f7abab879dd9: Status 404 returned error can't find the container with id da85dbb4c991180c35dcb8c1d0e0dc8eb47d251d6284a195d374f7abab879dd9
Sep 17 19:21:45.053600 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:45.053511    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" event={"ID":"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce","Type":"ContainerStarted","Data":"b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5"}
Sep 17 19:21:45.053600 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:45.053588    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:21:45.053789 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:45.053606    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" event={"ID":"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce","Type":"ContainerStarted","Data":"da85dbb4c991180c35dcb8c1d0e0dc8eb47d251d6284a195d374f7abab879dd9"}
Sep 17 19:21:45.114425 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:21:45.114366    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" podStartSLOduration=1.114348638 podStartE2EDuration="1.114348638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:21:45.108337977 +0000 UTC m=+2525.968162638" watchObservedRunningTime="2026-09-17 19:21:45.114348638 +0000 UTC m=+2525.974173286"
Sep 17 19:22:16.060470 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:22:16.060433    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:22:36.992763 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:22:36.992726    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"]
Sep 17 19:22:44.667983 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:22:44.667942    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9" podUID="121cf523-18bf-4ac8-99b6-0973d3768d44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:22:52.013815 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:22:52.013720    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9" podUID="121cf523-18bf-4ac8-99b6-0973d3768d44" containerName="main" containerID="cri-o://30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76" gracePeriod=45
Sep 17 19:22:54.668979 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:22:54.668936    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9" podUID="121cf523-18bf-4ac8-99b6-0973d3768d44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:23:04.668650 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:04.668608    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9" podUID="121cf523-18bf-4ac8-99b6-0973d3768d44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:23:04.669180 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:04.668717    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:23:14.668696 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:14.668657    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9" podUID="121cf523-18bf-4ac8-99b6-0973d3768d44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:23:22.268458 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.268429    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:23:22.347772 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.347731    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121cf523-18bf-4ac8-99b6-0973d3768d44-tls-certs\") pod \"121cf523-18bf-4ac8-99b6-0973d3768d44\" (UID: \"121cf523-18bf-4ac8-99b6-0973d3768d44\") "
Sep 17 19:23:22.350206 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.350173    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121cf523-18bf-4ac8-99b6-0973d3768d44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "121cf523-18bf-4ac8-99b6-0973d3768d44" (UID: "121cf523-18bf-4ac8-99b6-0973d3768d44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:23:22.448576 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.448539    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7hrb\" (UniqueName: \"kubernetes.io/projected/121cf523-18bf-4ac8-99b6-0973d3768d44-kube-api-access-d7hrb\") pod \"121cf523-18bf-4ac8-99b6-0973d3768d44\" (UID: \"121cf523-18bf-4ac8-99b6-0973d3768d44\") "
Sep 17 19:23:22.448749 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.448722    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121cf523-18bf-4ac8-99b6-0973d3768d44-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:23:22.450849 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.450802    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121cf523-18bf-4ac8-99b6-0973d3768d44-kube-api-access-d7hrb" (OuterVolumeSpecName: "kube-api-access-d7hrb") pod "121cf523-18bf-4ac8-99b6-0973d3768d44" (UID: "121cf523-18bf-4ac8-99b6-0973d3768d44"). InnerVolumeSpecName "kube-api-access-d7hrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:22.549268 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.549232    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7hrb\" (UniqueName: \"kubernetes.io/projected/121cf523-18bf-4ac8-99b6-0973d3768d44-kube-api-access-d7hrb\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:23:22.586439 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.586404    2586 generic.go:356] "Generic (PLEG): container finished" podID="121cf523-18bf-4ac8-99b6-0973d3768d44" containerID="30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76" exitCode=0
Sep 17 19:23:22.586616 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.586444    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9" event={"ID":"121cf523-18bf-4ac8-99b6-0973d3768d44","Type":"ContainerDied","Data":"30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76"}
Sep 17 19:23:22.586616 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.586482    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9" event={"ID":"121cf523-18bf-4ac8-99b6-0973d3768d44","Type":"ContainerDied","Data":"e4cdb3d9df67fb6e93725a23df03c45aee58b995c9d72951f4aead80e35a8bf9"}
Sep 17 19:23:22.586616 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.586502    2586 scope.go:117] "RemoveContainer" containerID="30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76"
Sep 17 19:23:22.586616 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.586448    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"
Sep 17 19:23:22.598415 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.597685    2586 scope.go:117] "RemoveContainer" containerID="30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76"
Sep 17 19:23:22.598415 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:23:22.598114    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76\": container with ID starting with 30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76 not found: ID does not exist" containerID="30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76"
Sep 17 19:23:22.598415 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.598149    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76"} err="failed to get container status \"30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76\": rpc error: code = NotFound desc = could not find container \"30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76\": container with ID starting with 30f1cfa8d7804dbb2f20e2b31c083866a806c3ca6a6b2fd2606286f4a3c34e76 not found: ID does not exist"
Sep 17 19:23:22.611591 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.611557    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"]
Sep 17 19:23:22.614981 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:22.614949    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mkdk9"]
Sep 17 19:23:23.458849 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:23:23.458803    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="121cf523-18bf-4ac8-99b6-0973d3768d44" path="/var/lib/kubelet/pods/121cf523-18bf-4ac8-99b6-0973d3768d44/volumes"
Sep 17 19:24:03.654510 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:03.654473    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"]
Sep 17 19:24:06.061321 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:06.061279    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" podUID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:24:16.061020 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:16.060975    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" podUID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:24:17.967700 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:17.967661    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"]
Sep 17 19:24:17.968269 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:17.968249    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121cf523-18bf-4ac8-99b6-0973d3768d44" containerName="main"
Sep 17 19:24:17.968357 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:17.968273    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="121cf523-18bf-4ac8-99b6-0973d3768d44" containerName="main"
Sep 17 19:24:17.968411 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:17.968386    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="121cf523-18bf-4ac8-99b6-0973d3768d44" containerName="main"
Sep 17 19:24:17.971606 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:17.971582    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:24:17.974212 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:17.974186    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-shk68\""
Sep 17 19:24:17.980014 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:17.979989    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"]
Sep 17 19:24:18.050233 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.050200    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7rb9\" (UniqueName: \"kubernetes.io/projected/dad9db43-c0e8-40e2-97c4-78da7ad57138-kube-api-access-q7rb9\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p\" (UID: \"dad9db43-c0e8-40e2-97c4-78da7ad57138\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:24:18.050429 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.050252    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad9db43-c0e8-40e2-97c4-78da7ad57138-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p\" (UID: \"dad9db43-c0e8-40e2-97c4-78da7ad57138\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:24:18.151192 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.151143    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7rb9\" (UniqueName: \"kubernetes.io/projected/dad9db43-c0e8-40e2-97c4-78da7ad57138-kube-api-access-q7rb9\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p\" (UID: \"dad9db43-c0e8-40e2-97c4-78da7ad57138\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:24:18.151192 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.151197    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad9db43-c0e8-40e2-97c4-78da7ad57138-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p\" (UID: \"dad9db43-c0e8-40e2-97c4-78da7ad57138\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:24:18.155291 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.155259    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad9db43-c0e8-40e2-97c4-78da7ad57138-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p\" (UID: \"dad9db43-c0e8-40e2-97c4-78da7ad57138\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:24:18.161476 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.161451    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7rb9\" (UniqueName: \"kubernetes.io/projected/dad9db43-c0e8-40e2-97c4-78da7ad57138-kube-api-access-q7rb9\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p\" (UID: \"dad9db43-c0e8-40e2-97c4-78da7ad57138\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:24:18.282816 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.282704    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:24:18.557211 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.557185    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"]
Sep 17 19:24:18.559260 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:24:18.559233    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad9db43_c0e8_40e2_97c4_78da7ad57138.slice/crio-f23e2cbb9d617b29566f46cb27a93c82f19d31f666835353a951fdc67b242475 WatchSource:0}: Error finding container f23e2cbb9d617b29566f46cb27a93c82f19d31f666835353a951fdc67b242475: Status 404 returned error can't find the container with id f23e2cbb9d617b29566f46cb27a93c82f19d31f666835353a951fdc67b242475
Sep 17 19:24:18.675632 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.675584    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" podUID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" containerName="main" containerID="cri-o://b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5" gracePeriod=45
Sep 17 19:24:18.898594 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.898551    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p" event={"ID":"dad9db43-c0e8-40e2-97c4-78da7ad57138","Type":"ContainerStarted","Data":"acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f"}
Sep 17 19:24:18.898801 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.898646    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:24:18.898801 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.898669    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p" event={"ID":"dad9db43-c0e8-40e2-97c4-78da7ad57138","Type":"ContainerStarted","Data":"f23e2cbb9d617b29566f46cb27a93c82f19d31f666835353a951fdc67b242475"}
Sep 17 19:24:18.918063 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:18.918002    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p" podStartSLOduration=1.917986796 podStartE2EDuration="1.917986796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:24:18.915552621 +0000 UTC m=+2679.775377289" watchObservedRunningTime="2026-09-17 19:24:18.917986796 +0000 UTC m=+2679.777811443"
Sep 17 19:24:26.061032 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:26.060945    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" podUID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:24:26.061426 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:26.061047    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:24:36.060588 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:36.060547    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" podUID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:24:41.624026 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:41.623998    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:24:41.635133 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:41.635106    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:24:46.061204 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:46.061161    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" podUID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:24:48.920729 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:48.920700    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:24:49.023770 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.023674    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-tls-certs\") pod \"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce\" (UID: \"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce\") "
Sep 17 19:24:49.023770 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.023737    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzkkk\" (UniqueName: \"kubernetes.io/projected/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-kube-api-access-xzkkk\") pod \"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce\" (UID: \"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce\") "
Sep 17 19:24:49.026256 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.026225    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" (UID: "a7d1d617-bd19-4f42-aa77-ac04dc7e5dce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:24:49.026349 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.026288    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-kube-api-access-xzkkk" (OuterVolumeSpecName: "kube-api-access-xzkkk") pod "a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" (UID: "a7d1d617-bd19-4f42-aa77-ac04dc7e5dce"). InnerVolumeSpecName "kube-api-access-xzkkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:24:49.066459 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.066423    2586 generic.go:356] "Generic (PLEG): container finished" podID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" containerID="b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5" exitCode=0
Sep 17 19:24:49.066669 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.066469    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"
Sep 17 19:24:49.066669 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.066514    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" event={"ID":"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce","Type":"ContainerDied","Data":"b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5"}
Sep 17 19:24:49.066669 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.066574    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v" event={"ID":"a7d1d617-bd19-4f42-aa77-ac04dc7e5dce","Type":"ContainerDied","Data":"da85dbb4c991180c35dcb8c1d0e0dc8eb47d251d6284a195d374f7abab879dd9"}
Sep 17 19:24:49.066669 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.066605    2586 scope.go:117] "RemoveContainer" containerID="b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5"
Sep 17 19:24:49.076648 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.076631    2586 scope.go:117] "RemoveContainer" containerID="b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5"
Sep 17 19:24:49.076987 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:24:49.076969    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5\": container with ID starting with b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5 not found: ID does not exist" containerID="b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5"
Sep 17 19:24:49.077030 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.076996    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5"} err="failed to get container status \"b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5\": rpc error: code = NotFound desc = could not find container \"b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5\": container with ID starting with b41a1732d8dfc09c8cd3d1a334e4c621bc2af837dc374ee5694b1e5e344e58b5 not found: ID does not exist"
Sep 17 19:24:49.091081 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.091008    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"]
Sep 17 19:24:49.093383 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.093358    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gxl4v"]
Sep 17 19:24:49.124750 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.124719    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:24:49.124750 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.124747    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzkkk\" (UniqueName: \"kubernetes.io/projected/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce-kube-api-access-xzkkk\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:24:49.459657 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.459610    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" path="/var/lib/kubelet/pods/a7d1d617-bd19-4f42-aa77-ac04dc7e5dce/volumes"
Sep 17 19:24:49.906570 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:24:49.906542    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:26:20.056223 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:26:20.056184    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"]
Sep 17 19:26:29.906209 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:26:29.906170    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p" podUID="dad9db43-c0e8-40e2-97c4-78da7ad57138" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:26:35.077479 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:26:35.077435    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p" podUID="dad9db43-c0e8-40e2-97c4-78da7ad57138" containerName="main" containerID="cri-o://acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f" gracePeriod=45
Sep 17 19:26:39.906211 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:26:39.906167    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p" podUID="dad9db43-c0e8-40e2-97c4-78da7ad57138" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:26:49.906227 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:26:49.906184    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p" podUID="dad9db43-c0e8-40e2-97c4-78da7ad57138" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:26:49.906738 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:26:49.906326    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:26:59.906365 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:26:59.906318    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p" podUID="dad9db43-c0e8-40e2-97c4-78da7ad57138" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:05.335022 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.334995    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:27:05.403005 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.402916    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad9db43-c0e8-40e2-97c4-78da7ad57138-tls-certs\") pod \"dad9db43-c0e8-40e2-97c4-78da7ad57138\" (UID: \"dad9db43-c0e8-40e2-97c4-78da7ad57138\") "
Sep 17 19:27:05.403162 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.403022    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7rb9\" (UniqueName: \"kubernetes.io/projected/dad9db43-c0e8-40e2-97c4-78da7ad57138-kube-api-access-q7rb9\") pod \"dad9db43-c0e8-40e2-97c4-78da7ad57138\" (UID: \"dad9db43-c0e8-40e2-97c4-78da7ad57138\") "
Sep 17 19:27:05.405267 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.405231    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad9db43-c0e8-40e2-97c4-78da7ad57138-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dad9db43-c0e8-40e2-97c4-78da7ad57138" (UID: "dad9db43-c0e8-40e2-97c4-78da7ad57138"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:05.405386 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.405240    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad9db43-c0e8-40e2-97c4-78da7ad57138-kube-api-access-q7rb9" (OuterVolumeSpecName: "kube-api-access-q7rb9") pod "dad9db43-c0e8-40e2-97c4-78da7ad57138" (UID: "dad9db43-c0e8-40e2-97c4-78da7ad57138"). InnerVolumeSpecName "kube-api-access-q7rb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:05.504360 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.504322    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7rb9\" (UniqueName: \"kubernetes.io/projected/dad9db43-c0e8-40e2-97c4-78da7ad57138-kube-api-access-q7rb9\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:27:05.504360 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.504354    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad9db43-c0e8-40e2-97c4-78da7ad57138-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:27:05.824700 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.824667    2586 generic.go:356] "Generic (PLEG): container finished" podID="dad9db43-c0e8-40e2-97c4-78da7ad57138" containerID="acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f" exitCode=0
Sep 17 19:27:05.824934 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.824716    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"
Sep 17 19:27:05.824934 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.824757    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p" event={"ID":"dad9db43-c0e8-40e2-97c4-78da7ad57138","Type":"ContainerDied","Data":"acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f"}
Sep 17 19:27:05.824934 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.824816    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p" event={"ID":"dad9db43-c0e8-40e2-97c4-78da7ad57138","Type":"ContainerDied","Data":"f23e2cbb9d617b29566f46cb27a93c82f19d31f666835353a951fdc67b242475"}
Sep 17 19:27:05.824934 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.824868    2586 scope.go:117] "RemoveContainer" containerID="acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f"
Sep 17 19:27:05.834423 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.834397    2586 scope.go:117] "RemoveContainer" containerID="acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f"
Sep 17 19:27:05.834755 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:27:05.834731    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f\": container with ID starting with acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f not found: ID does not exist" containerID="acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f"
Sep 17 19:27:05.834817 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.834769    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f"} err="failed to get container status \"acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f\": rpc error: code = NotFound desc = could not find container \"acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f\": container with ID starting with acc32d6cc8e3016e5bb2d515fc72af762fe1861fb04d2e00e98104c95eef471f not found: ID does not exist"
Sep 17 19:27:05.844454 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.844420    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"]
Sep 17 19:27:05.852171 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:05.852143    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-52m5p"]
Sep 17 19:27:07.459282 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:07.459247    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad9db43-c0e8-40e2-97c4-78da7ad57138" path="/var/lib/kubelet/pods/dad9db43-c0e8-40e2-97c4-78da7ad57138/volumes"
Sep 17 19:27:47.791300 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.791260    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"]
Sep 17 19:27:47.791931 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.791625    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" containerName="main"
Sep 17 19:27:47.791931 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.791636    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" containerName="main"
Sep 17 19:27:47.791931 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.791648    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad9db43-c0e8-40e2-97c4-78da7ad57138" containerName="main"
Sep 17 19:27:47.791931 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.791662    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad9db43-c0e8-40e2-97c4-78da7ad57138" containerName="main"
Sep 17 19:27:47.791931 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.791749    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d1d617-bd19-4f42-aa77-ac04dc7e5dce" containerName="main"
Sep 17 19:27:47.791931 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.791760    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad9db43-c0e8-40e2-97c4-78da7ad57138" containerName="main"
Sep 17 19:27:47.795182 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.795157    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:27:47.799159 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.799113    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-5zmls\""
Sep 17 19:27:47.799159 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.799133    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 19:27:47.799159 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.799156    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 19:27:47.799427 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.799113    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-9qhn6\""
Sep 17 19:27:47.799427 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.799229    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 19:27:47.804763 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.804735    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"]
Sep 17 19:27:47.986864 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.986803    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ac30fc7-ed45-459e-834a-bbf267eaae07-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r\" (UID: \"0ac30fc7-ed45-459e-834a-bbf267eaae07\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:27:47.986864 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:47.986858    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv5dd\" (UniqueName: \"kubernetes.io/projected/0ac30fc7-ed45-459e-834a-bbf267eaae07-kube-api-access-fv5dd\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r\" (UID: \"0ac30fc7-ed45-459e-834a-bbf267eaae07\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:27:48.087530 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:48.087486    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ac30fc7-ed45-459e-834a-bbf267eaae07-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r\" (UID: \"0ac30fc7-ed45-459e-834a-bbf267eaae07\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:27:48.087755 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:48.087546    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv5dd\" (UniqueName: \"kubernetes.io/projected/0ac30fc7-ed45-459e-834a-bbf267eaae07-kube-api-access-fv5dd\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r\" (UID: \"0ac30fc7-ed45-459e-834a-bbf267eaae07\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:27:48.090276 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:48.090256    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ac30fc7-ed45-459e-834a-bbf267eaae07-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r\" (UID: \"0ac30fc7-ed45-459e-834a-bbf267eaae07\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:27:48.096423 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:48.096370    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv5dd\" (UniqueName: \"kubernetes.io/projected/0ac30fc7-ed45-459e-834a-bbf267eaae07-kube-api-access-fv5dd\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r\" (UID: \"0ac30fc7-ed45-459e-834a-bbf267eaae07\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:27:48.107116 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:48.107087    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:27:48.386500 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:48.386465    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"]
Sep 17 19:27:48.387858 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:27:48.387807    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac30fc7_ed45_459e_834a_bbf267eaae07.slice/crio-456fa477f99dd64358a5397bf02435f0060c108c4fe65c8c7b999a215f107151 WatchSource:0}: Error finding container 456fa477f99dd64358a5397bf02435f0060c108c4fe65c8c7b999a215f107151: Status 404 returned error can't find the container with id 456fa477f99dd64358a5397bf02435f0060c108c4fe65c8c7b999a215f107151
Sep 17 19:27:49.063716 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:49.063678    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r" event={"ID":"0ac30fc7-ed45-459e-834a-bbf267eaae07","Type":"ContainerStarted","Data":"542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30"}
Sep 17 19:27:49.064168 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:49.063732    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r" event={"ID":"0ac30fc7-ed45-459e-834a-bbf267eaae07","Type":"ContainerStarted","Data":"456fa477f99dd64358a5397bf02435f0060c108c4fe65c8c7b999a215f107151"}
Sep 17 19:27:49.064168 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:49.063763    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:27:49.085759 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:27:49.085700    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r" podStartSLOduration=2.085682522 podStartE2EDuration="2.085682522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:49.084071758 +0000 UTC m=+2889.943896423" watchObservedRunningTime="2026-09-17 19:27:49.085682522 +0000 UTC m=+2889.945507170"
Sep 17 19:28:20.075143 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:28:20.075109    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:29:41.660789 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:29:41.660761    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:29:41.675066 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:29:41.675041    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:30:12.178426 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:12.178384    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"]
Sep 17 19:30:20.074783 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:20.074747    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r" podUID="0ac30fc7-ed45-459e-834a-bbf267eaae07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:27.202161 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:27.202119    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r" podUID="0ac30fc7-ed45-459e-834a-bbf267eaae07" containerName="main" containerID="cri-o://542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30" gracePeriod=45
Sep 17 19:30:30.074733 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:30.074691    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r" podUID="0ac30fc7-ed45-459e-834a-bbf267eaae07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:40.075245 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:40.075204    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r" podUID="0ac30fc7-ed45-459e-834a-bbf267eaae07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:40.075727 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:40.075319    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:30:50.075343 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:50.075299    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r" podUID="0ac30fc7-ed45-459e-834a-bbf267eaae07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:57.454963 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:57.454931    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:30:57.526269 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:57.526179    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv5dd\" (UniqueName: \"kubernetes.io/projected/0ac30fc7-ed45-459e-834a-bbf267eaae07-kube-api-access-fv5dd\") pod \"0ac30fc7-ed45-459e-834a-bbf267eaae07\" (UID: \"0ac30fc7-ed45-459e-834a-bbf267eaae07\") "
Sep 17 19:30:57.526269 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:57.526268    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ac30fc7-ed45-459e-834a-bbf267eaae07-tls-certs\") pod \"0ac30fc7-ed45-459e-834a-bbf267eaae07\" (UID: \"0ac30fc7-ed45-459e-834a-bbf267eaae07\") "
Sep 17 19:30:57.528638 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:57.528603    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac30fc7-ed45-459e-834a-bbf267eaae07-kube-api-access-fv5dd" (OuterVolumeSpecName: "kube-api-access-fv5dd") pod "0ac30fc7-ed45-459e-834a-bbf267eaae07" (UID: "0ac30fc7-ed45-459e-834a-bbf267eaae07"). InnerVolumeSpecName "kube-api-access-fv5dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:57.528754 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:57.528633    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac30fc7-ed45-459e-834a-bbf267eaae07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ac30fc7-ed45-459e-834a-bbf267eaae07" (UID: "0ac30fc7-ed45-459e-834a-bbf267eaae07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:57.627159 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:57.627121    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ac30fc7-ed45-459e-834a-bbf267eaae07-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:30:57.627159 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:57.627152    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv5dd\" (UniqueName: \"kubernetes.io/projected/0ac30fc7-ed45-459e-834a-bbf267eaae07-kube-api-access-fv5dd\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.091365 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:58.091329    2586 generic.go:356] "Generic (PLEG): container finished" podID="0ac30fc7-ed45-459e-834a-bbf267eaae07" containerID="542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30" exitCode=0
Sep 17 19:30:58.091577 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:58.091367    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"
Sep 17 19:30:58.091577 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:58.091415    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r" event={"ID":"0ac30fc7-ed45-459e-834a-bbf267eaae07","Type":"ContainerDied","Data":"542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30"}
Sep 17 19:30:58.091577 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:58.091464    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r" event={"ID":"0ac30fc7-ed45-459e-834a-bbf267eaae07","Type":"ContainerDied","Data":"456fa477f99dd64358a5397bf02435f0060c108c4fe65c8c7b999a215f107151"}
Sep 17 19:30:58.091577 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:58.091486    2586 scope.go:117] "RemoveContainer" containerID="542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30"
Sep 17 19:30:58.101157 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:58.101139    2586 scope.go:117] "RemoveContainer" containerID="542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30"
Sep 17 19:30:58.101445 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:30:58.101426    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30\": container with ID starting with 542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30 not found: ID does not exist" containerID="542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30"
Sep 17 19:30:58.101494 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:58.101457    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30"} err="failed to get container status \"542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30\": rpc error: code = NotFound desc = could not find container \"542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30\": container with ID starting with 542efa7efee382bb824b12e540307b5877454dc8c0e3e07dddb60dc1acecaa30 not found: ID does not exist"
Sep 17 19:30:58.114731 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:58.114699    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"]
Sep 17 19:30:58.118814 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:58.118787    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jsf8r"]
Sep 17 19:30:59.460060 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:30:59.460001    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ac30fc7-ed45-459e-834a-bbf267eaae07" path="/var/lib/kubelet/pods/0ac30fc7-ed45-459e-834a-bbf267eaae07/volumes"
Sep 17 19:31:11.869043 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.869003    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"]
Sep 17 19:31:11.869715 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.869501    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac30fc7-ed45-459e-834a-bbf267eaae07" containerName="main"
Sep 17 19:31:11.869715 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.869524    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac30fc7-ed45-459e-834a-bbf267eaae07" containerName="main"
Sep 17 19:31:11.869715 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.869648    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ac30fc7-ed45-459e-834a-bbf267eaae07" containerName="main"
Sep 17 19:31:11.872621 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.872601    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:31:11.876425 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.876381    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 19:31:11.876573 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.876380    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-w6j47\""
Sep 17 19:31:11.876573 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.876471    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-lwnjk\""
Sep 17 19:31:11.876573 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.876380    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 19:31:11.876573 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.876552    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 19:31:11.889110 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.889064    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"]
Sep 17 19:31:11.953026 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.952981    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqsbf\" (UniqueName: \"kubernetes.io/projected/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-kube-api-access-nqsbf\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz\" (UID: \"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:31:11.953026 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:11.953030    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz\" (UID: \"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:31:12.053846 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:12.053789    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqsbf\" (UniqueName: \"kubernetes.io/projected/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-kube-api-access-nqsbf\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz\" (UID: \"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:31:12.054018 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:12.053890    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz\" (UID: \"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:31:12.057045 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:12.057017    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz\" (UID: \"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:31:12.066649 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:12.066619    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqsbf\" (UniqueName: \"kubernetes.io/projected/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-kube-api-access-nqsbf\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz\" (UID: \"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:31:12.185744 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:12.185646    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:31:12.461706 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:12.461677    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"]
Sep 17 19:31:12.463299 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:31:12.463267    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbd1d2b_670d_4b1f_88a7_a0af00b8b861.slice/crio-e7af61ff8645d0860d67b0c482b3eb8baa11a7dce38ae46e569738fa2fdb9ee6 WatchSource:0}: Error finding container e7af61ff8645d0860d67b0c482b3eb8baa11a7dce38ae46e569738fa2fdb9ee6: Status 404 returned error can't find the container with id e7af61ff8645d0860d67b0c482b3eb8baa11a7dce38ae46e569738fa2fdb9ee6
Sep 17 19:31:13.180950 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:13.180916    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" event={"ID":"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861","Type":"ContainerStarted","Data":"db21e880e2adef7bd161ccc825e3c0c87cc031713743480f55f1a0215f56f7ec"}
Sep 17 19:31:13.181385 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:13.180967    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" event={"ID":"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861","Type":"ContainerStarted","Data":"e7af61ff8645d0860d67b0c482b3eb8baa11a7dce38ae46e569738fa2fdb9ee6"}
Sep 17 19:31:13.181385 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:13.181051    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:31:13.199078 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:13.199019    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" podStartSLOduration=2.199000873 podStartE2EDuration="2.199000873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:13.198184581 +0000 UTC m=+3094.058009253" watchObservedRunningTime="2026-09-17 19:31:13.199000873 +0000 UTC m=+3094.058825558"
Sep 17 19:31:44.188747 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:31:44.188714    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:33:31.005353 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:33:31.005261    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"]
Sep 17 19:33:34.188513 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:33:34.188476    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" podUID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:33:44.187955 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:33:44.187912    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" podUID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:33:46.025221 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:33:46.025177    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" podUID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" containerName="main" containerID="cri-o://db21e880e2adef7bd161ccc825e3c0c87cc031713743480f55f1a0215f56f7ec" gracePeriod=45
Sep 17 19:33:54.188670 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:33:54.188631    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" podUID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:33:54.189290 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:33:54.188777    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:34:04.188370 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:04.188329    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" podUID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:34:14.188669 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:14.188627    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" podUID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:34:16.175908 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:16.175874    2586 generic.go:356] "Generic (PLEG): container finished" podID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" containerID="db21e880e2adef7bd161ccc825e3c0c87cc031713743480f55f1a0215f56f7ec" exitCode=0
Sep 17 19:34:16.176309 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:16.175952    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" event={"ID":"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861","Type":"ContainerDied","Data":"db21e880e2adef7bd161ccc825e3c0c87cc031713743480f55f1a0215f56f7ec"}
Sep 17 19:34:16.279593 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:16.279566    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:34:16.358296 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:16.358258    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqsbf\" (UniqueName: \"kubernetes.io/projected/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-kube-api-access-nqsbf\") pod \"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861\" (UID: \"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861\") "
Sep 17 19:34:16.358501 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:16.358399    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-tls-certs\") pod \"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861\" (UID: \"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861\") "
Sep 17 19:34:16.360690 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:16.360656    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-kube-api-access-nqsbf" (OuterVolumeSpecName: "kube-api-access-nqsbf") pod "2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" (UID: "2dbd1d2b-670d-4b1f-88a7-a0af00b8b861"). InnerVolumeSpecName "kube-api-access-nqsbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:16.360814 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:16.360721    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" (UID: "2dbd1d2b-670d-4b1f-88a7-a0af00b8b861"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:34:16.459782 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:16.459742    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:34:16.459782 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:16.459773    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqsbf\" (UniqueName: \"kubernetes.io/projected/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861-kube-api-access-nqsbf\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:34:17.183571 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:17.183541    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"
Sep 17 19:34:17.184064 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:17.183584    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz" event={"ID":"2dbd1d2b-670d-4b1f-88a7-a0af00b8b861","Type":"ContainerDied","Data":"e7af61ff8645d0860d67b0c482b3eb8baa11a7dce38ae46e569738fa2fdb9ee6"}
Sep 17 19:34:17.184064 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:17.183645    2586 scope.go:117] "RemoveContainer" containerID="db21e880e2adef7bd161ccc825e3c0c87cc031713743480f55f1a0215f56f7ec"
Sep 17 19:34:17.208311 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:17.208277    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"]
Sep 17 19:34:17.213713 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:17.213686    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45bc6hz"]
Sep 17 19:34:17.458030 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:17.457950    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" path="/var/lib/kubelet/pods/2dbd1d2b-670d-4b1f-88a7-a0af00b8b861/volumes"
Sep 17 19:34:41.697917 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:41.697884    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:34:41.716158 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:41.716132    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:34:56.690108 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.689996    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"]
Sep 17 19:34:56.690544 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.690490    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" containerName="main"
Sep 17 19:34:56.690544 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.690505    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" containerName="main"
Sep 17 19:34:56.690638 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.690573    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbd1d2b-670d-4b1f-88a7-a0af00b8b861" containerName="main"
Sep 17 19:34:56.695202 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.695181    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.697981 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.697956    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 19:34:56.698132 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.698060    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-k7j6h\""
Sep 17 19:34:56.698257 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.698239    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 19:34:56.698340 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.698324    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 19:34:56.707303 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.707271    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"]
Sep 17 19:34:56.811810 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.811779    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z75t\" (UniqueName: \"kubernetes.io/projected/9ab87150-47ce-456b-8a03-ce208adbd2bf-kube-api-access-9z75t\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.811810 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.811818    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab87150-47ce-456b-8a03-ce208adbd2bf-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.812051 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.811917    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.812051 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.811968    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.812051 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.811985    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.812051 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.812043    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.913431 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.913388    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.913431 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.913439    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z75t\" (UniqueName: \"kubernetes.io/projected/9ab87150-47ce-456b-8a03-ce208adbd2bf-kube-api-access-9z75t\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.913712 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.913464    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab87150-47ce-456b-8a03-ce208adbd2bf-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.913712 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.913498    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.913712 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.913530    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.913712 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.913547    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.913938 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.913923    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.913989 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.913925    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.914038 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.913996    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.916046 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.916018    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.916331 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.916315    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab87150-47ce-456b-8a03-ce208adbd2bf-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:56.922228 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:56.922207    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z75t\" (UniqueName: \"kubernetes.io/projected/9ab87150-47ce-456b-8a03-ce208adbd2bf-kube-api-access-9z75t\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:59.032769 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:59.032730    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:59.041760 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:59.041731    2586 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:59.133620 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:59.133580    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:59.133821 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:59.133727    2586 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:59.141605 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:59.141576    2586 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") DevicePath \"csi-2f3445d44ce68e69aa38423a5e8aa00d0cdbc096526373cac2a915ffc931e926\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:59.241691 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:59.241662    2586 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/e7320d8a547ca060c7b87daa9748e7e7439b749410ef06b584b42b4d24b8df4c/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:59.267932 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:59.267899    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:59.407822 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:59.407780    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:34:59.690687 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:59.690632    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"]
Sep 17 19:34:59.693149 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:34:59.693113    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab87150_47ce_456b_8a03_ce208adbd2bf.slice/crio-0a2ff30247f890008d8f657415d07bb897d003e5a6758ebfde7c56f4b1e2311f WatchSource:0}: Error finding container 0a2ff30247f890008d8f657415d07bb897d003e5a6758ebfde7c56f4b1e2311f: Status 404 returned error can't find the container with id 0a2ff30247f890008d8f657415d07bb897d003e5a6758ebfde7c56f4b1e2311f
Sep 17 19:34:59.695157 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:34:59.695138    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:35:00.432704 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:35:00.432663    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" event={"ID":"9ab87150-47ce-456b-8a03-ce208adbd2bf","Type":"ContainerStarted","Data":"0a2ff30247f890008d8f657415d07bb897d003e5a6758ebfde7c56f4b1e2311f"}
Sep 17 19:35:28.630300 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:35:28.630254    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" event={"ID":"9ab87150-47ce-456b-8a03-ce208adbd2bf","Type":"ContainerStarted","Data":"8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10"}
Sep 17 19:35:28.667331 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:35:28.667262    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podStartSLOduration=4.78286485 podStartE2EDuration="32.667238484s" totalImagesPullingTime="27.884373634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:35:28.660063921 +0000 UTC m=+3349.519888585" watchObservedRunningTime="2026-09-17 19:35:28.667238484 +0000 UTC m=+3349.527063133"
Sep 17 19:35:29.408405 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:35:29.408360    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:35:29.408604 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:35:29.408446    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:35:29.410093 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:35:29.410062    2586 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 19:35:39.408596 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:35:39.408538    2586 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 19:35:49.408635 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:35:49.408590    2586 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 19:35:59.408397 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:35:59.408347    2586 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 19:36:09.409093 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:36:09.409041    2586 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 19:36:19.409094 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:36:19.409042    2586 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 19:36:29.408974 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:36:29.408862    2586 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 19:36:39.408946 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:36:39.408888    2586 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 19:36:49.418483 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:36:49.418449    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:36:49.426729 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:36:49.426696    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:37:17.859732 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:17.859692    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s_9ab87150-47ce-456b-8a03-ce208adbd2bf/main/0.log"
Sep 17 19:37:19.099124 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:19.099089    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"]
Sep 17 19:37:34.122195 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.122143    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main" containerID="cri-o://8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10" gracePeriod=45
Sep 17 19:37:34.419954 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.419860    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 19:37:34.782582 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.782552    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:37:34.858961 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.858921    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-tmp-dir\") pod \"9ab87150-47ce-456b-8a03-ce208adbd2bf\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") "
Sep 17 19:37:34.859155 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.858980    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-home\") pod \"9ab87150-47ce-456b-8a03-ce208adbd2bf\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") "
Sep 17 19:37:34.859155 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.859102    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"9ab87150-47ce-456b-8a03-ce208adbd2bf\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") "
Sep 17 19:37:34.859155 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.859136    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z75t\" (UniqueName: \"kubernetes.io/projected/9ab87150-47ce-456b-8a03-ce208adbd2bf-kube-api-access-9z75t\") pod \"9ab87150-47ce-456b-8a03-ce208adbd2bf\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") "
Sep 17 19:37:34.859321 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.859166    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab87150-47ce-456b-8a03-ce208adbd2bf-tls-certs\") pod \"9ab87150-47ce-456b-8a03-ce208adbd2bf\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") "
Sep 17 19:37:34.859321 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.859194    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-model-cache\") pod \"9ab87150-47ce-456b-8a03-ce208adbd2bf\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") "
Sep 17 19:37:34.859321 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.859225    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-dshm\") pod \"9ab87150-47ce-456b-8a03-ce208adbd2bf\" (UID: \"9ab87150-47ce-456b-8a03-ce208adbd2bf\") "
Sep 17 19:37:34.859475 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.859367    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-home" (OuterVolumeSpecName: "home") pod "9ab87150-47ce-456b-8a03-ce208adbd2bf" (UID: "9ab87150-47ce-456b-8a03-ce208adbd2bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:34.859475 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.859423    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ab87150-47ce-456b-8a03-ce208adbd2bf" (UID: "9ab87150-47ce-456b-8a03-ce208adbd2bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:34.859586 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.859556    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:37:34.859586 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.859575    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:37:34.861650 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.861613    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-dshm" (OuterVolumeSpecName: "dshm") pod "9ab87150-47ce-456b-8a03-ce208adbd2bf" (UID: "9ab87150-47ce-456b-8a03-ce208adbd2bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:34.861976 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.861946    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab87150-47ce-456b-8a03-ce208adbd2bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ab87150-47ce-456b-8a03-ce208adbd2bf" (UID: "9ab87150-47ce-456b-8a03-ce208adbd2bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:37:34.862077 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.861995    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab87150-47ce-456b-8a03-ce208adbd2bf-kube-api-access-9z75t" (OuterVolumeSpecName: "kube-api-access-9z75t") pod "9ab87150-47ce-456b-8a03-ce208adbd2bf" (UID: "9ab87150-47ce-456b-8a03-ce208adbd2bf"). InnerVolumeSpecName "kube-api-access-9z75t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:37:34.868793 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.868772    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef" (OuterVolumeSpecName: "kserve-pvc-source") pod "9ab87150-47ce-456b-8a03-ce208adbd2bf" (UID: "9ab87150-47ce-456b-8a03-ce208adbd2bf"). InnerVolumeSpecName "pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:37:34.875820 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.875787    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ab87150-47ce-456b-8a03-ce208adbd2bf" (UID: "9ab87150-47ce-456b-8a03-ce208adbd2bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:34.960806 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.960705    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:37:34.960806 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.960756    2586 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") on node \"ip-10-0-129-204.ec2.internal\" "
Sep 17 19:37:34.960806 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.960767    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z75t\" (UniqueName: \"kubernetes.io/projected/9ab87150-47ce-456b-8a03-ce208adbd2bf-kube-api-access-9z75t\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:37:34.960806 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.960776    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab87150-47ce-456b-8a03-ce208adbd2bf-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:37:34.960806 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.960784    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab87150-47ce-456b-8a03-ce208adbd2bf-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:37:34.989792 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:34.989771    2586 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef") on node "ip-10-0-129-204.ec2.internal"
Sep 17 19:37:35.062153 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.062115    2586 reconciler_common.go:299] "Volume detached for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"csi-2f3445d44ce68e69aa38423a5e8aa00d0cdbc096526373cac2a915ffc931e926\""
Sep 17 19:37:35.385938 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.385905    2586 generic.go:356] "Generic (PLEG): container finished" podID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerID="8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10" exitCode=0
Sep 17 19:37:35.386376 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.385963    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"
Sep 17 19:37:35.386376 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.386004    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" event={"ID":"9ab87150-47ce-456b-8a03-ce208adbd2bf","Type":"ContainerDied","Data":"8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10"}
Sep 17 19:37:35.386376 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.386088    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s" event={"ID":"9ab87150-47ce-456b-8a03-ce208adbd2bf","Type":"ContainerDied","Data":"0a2ff30247f890008d8f657415d07bb897d003e5a6758ebfde7c56f4b1e2311f"}
Sep 17 19:37:35.386376 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.386111    2586 scope.go:117] "RemoveContainer" containerID="8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10"
Sep 17 19:37:35.396144 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.396122    2586 scope.go:117] "RemoveContainer" containerID="8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10"
Sep 17 19:37:35.396429 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:37:35.396412    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10\": container with ID starting with 8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10 not found: ID does not exist" containerID="8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10"
Sep 17 19:37:35.396478 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.396438    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10"} err="failed to get container status \"8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10\": rpc error: code = NotFound desc = could not find container \"8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10\": container with ID starting with 8292c8524ba6bb98d70b02098a580507e7f163d646be412d50a5fa692c2a9f10 not found: ID does not exist"
Sep 17 19:37:35.408512 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.408481    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"]
Sep 17 19:37:35.413250 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.413218    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-r2k7s"]
Sep 17 19:37:35.459552 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:37:35.459520    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" path="/var/lib/kubelet/pods/9ab87150-47ce-456b-8a03-ce208adbd2bf/volumes"
Sep 17 19:39:41.734644 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:39:41.734616    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:39:41.754576 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:39:41.754551    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:40:44.397293 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.397253    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"]
Sep 17 19:40:44.397796 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.397648    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main"
Sep 17 19:40:44.397796 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.397660    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main"
Sep 17 19:40:44.397796 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.397746    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab87150-47ce-456b-8a03-ce208adbd2bf" containerName="main"
Sep 17 19:40:44.400397 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.400377    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.403202 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.403177    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 19:40:44.403360 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.403344    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-cr7tr\""
Sep 17 19:40:44.404295 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.404270    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 19:40:44.404295 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.404291    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 19:40:44.413824 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.413705    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"]
Sep 17 19:40:44.492294 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.492255    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.492495 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.492304    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.492495 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.492445    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.492613 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.492538    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.492613 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.492580    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkn9p\" (UniqueName: \"kubernetes.io/projected/dfec046f-3490-4d6b-b5d8-13b4716b0598-kube-api-access-nkn9p\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.492721 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.492696    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfec046f-3490-4d6b-b5d8-13b4716b0598-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.593516 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.593477    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkn9p\" (UniqueName: \"kubernetes.io/projected/dfec046f-3490-4d6b-b5d8-13b4716b0598-kube-api-access-nkn9p\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.593742 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.593556    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfec046f-3490-4d6b-b5d8-13b4716b0598-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.593742 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.593725    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.593913 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.593776    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.594003 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.593908    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.594085 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.594069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.594266 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.594237    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.594397 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.594299    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.594397 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.594376    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.596609 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.596582    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfec046f-3490-4d6b-b5d8-13b4716b0598-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.596733 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.596651    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.609865 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.609816    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkn9p\" (UniqueName: \"kubernetes.io/projected/dfec046f-3490-4d6b-b5d8-13b4716b0598-kube-api-access-nkn9p\") pod \"scheduler-inline-config-test-kserve-8655bc757-ccjns\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.710823 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.710726    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:44.990220 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:44.990192    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"]
Sep 17 19:40:44.992046 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:40:44.992016    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfec046f_3490_4d6b_b5d8_13b4716b0598.slice/crio-78cc7098e6bff9aef6d50ad7158f77e43d0927aaf175adb8a341d1a6ab86fa3a WatchSource:0}: Error finding container 78cc7098e6bff9aef6d50ad7158f77e43d0927aaf175adb8a341d1a6ab86fa3a: Status 404 returned error can't find the container with id 78cc7098e6bff9aef6d50ad7158f77e43d0927aaf175adb8a341d1a6ab86fa3a
Sep 17 19:40:45.423590 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:45.423552    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns" event={"ID":"dfec046f-3490-4d6b-b5d8-13b4716b0598","Type":"ContainerStarted","Data":"ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6"}
Sep 17 19:40:45.424049 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:45.423612    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns" event={"ID":"dfec046f-3490-4d6b-b5d8-13b4716b0598","Type":"ContainerStarted","Data":"78cc7098e6bff9aef6d50ad7158f77e43d0927aaf175adb8a341d1a6ab86fa3a"}
Sep 17 19:40:45.469336 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:45.469277    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns" podStartSLOduration=1.469261486 podStartE2EDuration="1.469261486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:40:45.468534141 +0000 UTC m=+3666.328358817" watchObservedRunningTime="2026-09-17 19:40:45.469261486 +0000 UTC m=+3666.329086134"
Sep 17 19:40:45.711787 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:45.711698    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:54.711391 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:54.711295    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:54.724117 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:54.724087    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:40:55.493899 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:40:55.493868    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:41:18.741483 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:18.741442    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"]
Sep 17 19:41:18.753368 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:41:18.753315    2586 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:41:18.753368 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:41:18.753368 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:41:18.753368 ip-10-0-129-204 kubenswrapper[2586]:  > containerID="ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6" cmd=["/bin/sleep","15"]
Sep 17 19:41:18.753757 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:18.753379    2586 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns" message="" err=<
Sep 17 19:41:18.753757 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:41:18.753757 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:41:18.753757 ip-10-0-129-204 kubenswrapper[2586]:  >
Sep 17 19:41:18.753757 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:41:18.753397    2586 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:41:18.753757 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:41:18.753757 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:41:18.753757 ip-10-0-129-204 kubenswrapper[2586]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns" podUID="dfec046f-3490-4d6b-b5d8-13b4716b0598" containerName="main" containerID="cri-o://ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6"
Sep 17 19:41:18.753757 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:18.753440    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns" podUID="dfec046f-3490-4d6b-b5d8-13b4716b0598" containerName="main" containerID="cri-o://ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6" gracePeriod=60
Sep 17 19:41:19.112594 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.112563    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:41:19.200100 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200063    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-tmp-dir\") pod \"dfec046f-3490-4d6b-b5d8-13b4716b0598\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") "
Sep 17 19:41:19.200325 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200120    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkn9p\" (UniqueName: \"kubernetes.io/projected/dfec046f-3490-4d6b-b5d8-13b4716b0598-kube-api-access-nkn9p\") pod \"dfec046f-3490-4d6b-b5d8-13b4716b0598\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") "
Sep 17 19:41:19.200325 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200141    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-home\") pod \"dfec046f-3490-4d6b-b5d8-13b4716b0598\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") "
Sep 17 19:41:19.200325 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200159    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-model-cache\") pod \"dfec046f-3490-4d6b-b5d8-13b4716b0598\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") "
Sep 17 19:41:19.200325 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200206    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-dshm\") pod \"dfec046f-3490-4d6b-b5d8-13b4716b0598\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") "
Sep 17 19:41:19.200325 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200271    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfec046f-3490-4d6b-b5d8-13b4716b0598-tls-certs\") pod \"dfec046f-3490-4d6b-b5d8-13b4716b0598\" (UID: \"dfec046f-3490-4d6b-b5d8-13b4716b0598\") "
Sep 17 19:41:19.200582 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200386    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfec046f-3490-4d6b-b5d8-13b4716b0598" (UID: "dfec046f-3490-4d6b-b5d8-13b4716b0598"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:19.200582 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200404    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-home" (OuterVolumeSpecName: "home") pod "dfec046f-3490-4d6b-b5d8-13b4716b0598" (UID: "dfec046f-3490-4d6b-b5d8-13b4716b0598"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:19.200582 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200503    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfec046f-3490-4d6b-b5d8-13b4716b0598" (UID: "dfec046f-3490-4d6b-b5d8-13b4716b0598"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:19.200582 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200566    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.200724 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200587    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.200724 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.200604    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.202645 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.202606    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-dshm" (OuterVolumeSpecName: "dshm") pod "dfec046f-3490-4d6b-b5d8-13b4716b0598" (UID: "dfec046f-3490-4d6b-b5d8-13b4716b0598"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:19.202761 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.202675    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfec046f-3490-4d6b-b5d8-13b4716b0598-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfec046f-3490-4d6b-b5d8-13b4716b0598" (UID: "dfec046f-3490-4d6b-b5d8-13b4716b0598"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:41:19.202761 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.202688    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfec046f-3490-4d6b-b5d8-13b4716b0598-kube-api-access-nkn9p" (OuterVolumeSpecName: "kube-api-access-nkn9p") pod "dfec046f-3490-4d6b-b5d8-13b4716b0598" (UID: "dfec046f-3490-4d6b-b5d8-13b4716b0598"). InnerVolumeSpecName "kube-api-access-nkn9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:41:19.301372 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.301276    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfec046f-3490-4d6b-b5d8-13b4716b0598-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.301372 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.301311    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkn9p\" (UniqueName: \"kubernetes.io/projected/dfec046f-3490-4d6b-b5d8-13b4716b0598-kube-api-access-nkn9p\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.301372 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.301322    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfec046f-3490-4d6b-b5d8-13b4716b0598-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.618704 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.618669    2586 generic.go:356] "Generic (PLEG): container finished" podID="dfec046f-3490-4d6b-b5d8-13b4716b0598" containerID="ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6" exitCode=0
Sep 17 19:41:19.618944 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.618717    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"
Sep 17 19:41:19.618944 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.618751    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns" event={"ID":"dfec046f-3490-4d6b-b5d8-13b4716b0598","Type":"ContainerDied","Data":"ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6"}
Sep 17 19:41:19.618944 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.618799    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns" event={"ID":"dfec046f-3490-4d6b-b5d8-13b4716b0598","Type":"ContainerDied","Data":"78cc7098e6bff9aef6d50ad7158f77e43d0927aaf175adb8a341d1a6ab86fa3a"}
Sep 17 19:41:19.618944 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.618824    2586 scope.go:117] "RemoveContainer" containerID="ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6"
Sep 17 19:41:19.628714 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.628694    2586 scope.go:117] "RemoveContainer" containerID="ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6"
Sep 17 19:41:19.629074 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:41:19.629055    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6\": container with ID starting with ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6 not found: ID does not exist" containerID="ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6"
Sep 17 19:41:19.629149 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.629083    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6"} err="failed to get container status \"ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6\": rpc error: code = NotFound desc = could not find container \"ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6\": container with ID starting with ee5b3b5bf6b3b7bcdd29fab3e6d56673efcac1e0ab23edc91d89504e60f80da6 not found: ID does not exist"
Sep 17 19:41:19.635822 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.635788    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"]
Sep 17 19:41:19.639880 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:19.639855    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ccjns"]
Sep 17 19:41:21.458592 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:41:21.458559    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfec046f-3490-4d6b-b5d8-13b4716b0598" path="/var/lib/kubelet/pods/dfec046f-3490-4d6b-b5d8-13b4716b0598/volumes"
Sep 17 19:44:16.074887 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.074853    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"]
Sep 17 19:44:16.075333 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.075229    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfec046f-3490-4d6b-b5d8-13b4716b0598" containerName="main"
Sep 17 19:44:16.075333 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.075240    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfec046f-3490-4d6b-b5d8-13b4716b0598" containerName="main"
Sep 17 19:44:16.075333 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.075327    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfec046f-3490-4d6b-b5d8-13b4716b0598" containerName="main"
Sep 17 19:44:16.078387 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.078365    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.081183 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.081157    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 19:44:16.082332 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.082307    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-tplz6\""
Sep 17 19:44:16.082332 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.082322    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 19:44:16.082532 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.082393    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 19:44:16.089094 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.089065    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"]
Sep 17 19:44:16.181430 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.181390    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.181635 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.181446    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8h2f\" (UniqueName: \"kubernetes.io/projected/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-kube-api-access-z8h2f\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.181635 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.181534    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.181635 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.181591    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.181635 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.181615    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.181798 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.181640    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.283143 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.283108    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.283356 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.283173    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h2f\" (UniqueName: \"kubernetes.io/projected/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-kube-api-access-z8h2f\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.283356 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.283213    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.283356 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.283261    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.283356 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.283291    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.283356 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.283322    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.283875 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.283819    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.284007 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.283877    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.284007 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.283941    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.285945 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.285918    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.286203 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.286184    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.292662 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.292634    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8h2f\" (UniqueName: \"kubernetes.io/projected/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-kube-api-access-z8h2f\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.390626 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.390526    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:16.664982 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:16.664957    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"]
Sep 17 19:44:16.666572 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:44:16.666544    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9055893_3c29_4ff9_b0e6_8c0f9c84117d.slice/crio-2a1e5a7ac0e56b8e1eab86e3c428bf3b3ac77a7133bc8ebfb7a939f853f0b30d WatchSource:0}: Error finding container 2a1e5a7ac0e56b8e1eab86e3c428bf3b3ac77a7133bc8ebfb7a939f853f0b30d: Status 404 returned error can't find the container with id 2a1e5a7ac0e56b8e1eab86e3c428bf3b3ac77a7133bc8ebfb7a939f853f0b30d
Sep 17 19:44:17.595659 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:17.595615    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr" event={"ID":"b9055893-3c29-4ff9-b0e6-8c0f9c84117d","Type":"ContainerStarted","Data":"70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4"}
Sep 17 19:44:17.596089 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:17.595682    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr" event={"ID":"b9055893-3c29-4ff9-b0e6-8c0f9c84117d","Type":"ContainerStarted","Data":"2a1e5a7ac0e56b8e1eab86e3c428bf3b3ac77a7133bc8ebfb7a939f853f0b30d"}
Sep 17 19:44:17.619627 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:17.619572    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr" podStartSLOduration=1.619553579 podStartE2EDuration="1.619553579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:44:17.617136494 +0000 UTC m=+3878.476961162" watchObservedRunningTime="2026-09-17 19:44:17.619553579 +0000 UTC m=+3878.479378227"
Sep 17 19:44:18.391624 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:18.391579    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:26.391692 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:26.391638    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:26.404768 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:26.404736    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:26.656709 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:26.656618    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:41.773735 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:41.773706    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:44:41.803171 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:41.803142    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:44:50.090204 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.090169    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"]
Sep 17 19:44:50.099815 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:44:50.099773    2586 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:44:50.099815 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:44:50.099815 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:44:50.099815 ip-10-0-129-204 kubenswrapper[2586]:  > containerID="70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4" cmd=["/bin/sleep","15"]
Sep 17 19:44:50.100116 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.099856    2586 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr" message="" err=<
Sep 17 19:44:50.100116 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:44:50.100116 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:44:50.100116 ip-10-0-129-204 kubenswrapper[2586]:  >
Sep 17 19:44:50.100116 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:44:50.099876    2586 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:44:50.100116 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:44:50.100116 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:44:50.100116 ip-10-0-129-204 kubenswrapper[2586]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr" podUID="b9055893-3c29-4ff9-b0e6-8c0f9c84117d" containerName="main" containerID="cri-o://70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4"
Sep 17 19:44:50.100116 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.099920    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr" podUID="b9055893-3c29-4ff9-b0e6-8c0f9c84117d" containerName="main" containerID="cri-o://70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4" gracePeriod=60
Sep 17 19:44:50.451570 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.451543    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:50.486492 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.486451    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-home\") pod \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") "
Sep 17 19:44:50.486701 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.486512    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-dshm\") pod \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") "
Sep 17 19:44:50.486701 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.486545    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-model-cache\") pod \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") "
Sep 17 19:44:50.486701 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.486583    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tls-certs\") pod \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") "
Sep 17 19:44:50.486701 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.486619    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8h2f\" (UniqueName: \"kubernetes.io/projected/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-kube-api-access-z8h2f\") pod \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") "
Sep 17 19:44:50.486701 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.486670    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tmp-dir\") pod \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\" (UID: \"b9055893-3c29-4ff9-b0e6-8c0f9c84117d\") "
Sep 17 19:44:50.487008 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.486786    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-home" (OuterVolumeSpecName: "home") pod "b9055893-3c29-4ff9-b0e6-8c0f9c84117d" (UID: "b9055893-3c29-4ff9-b0e6-8c0f9c84117d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:50.487008 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.486854    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9055893-3c29-4ff9-b0e6-8c0f9c84117d" (UID: "b9055893-3c29-4ff9-b0e6-8c0f9c84117d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:50.487188 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.487142    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9055893-3c29-4ff9-b0e6-8c0f9c84117d" (UID: "b9055893-3c29-4ff9-b0e6-8c0f9c84117d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:50.487487 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.487404    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:44:50.487487 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.487428    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:44:50.487487 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.487443    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:44:50.489268 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.489237    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9055893-3c29-4ff9-b0e6-8c0f9c84117d" (UID: "b9055893-3c29-4ff9-b0e6-8c0f9c84117d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:44:50.489474 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.489451    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-dshm" (OuterVolumeSpecName: "dshm") pod "b9055893-3c29-4ff9-b0e6-8c0f9c84117d" (UID: "b9055893-3c29-4ff9-b0e6-8c0f9c84117d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:50.489594 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.489486    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-kube-api-access-z8h2f" (OuterVolumeSpecName: "kube-api-access-z8h2f") pod "b9055893-3c29-4ff9-b0e6-8c0f9c84117d" (UID: "b9055893-3c29-4ff9-b0e6-8c0f9c84117d"). InnerVolumeSpecName "kube-api-access-z8h2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:44:50.588248 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.588201    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:44:50.588248 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.588238    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:44:50.588248 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.588249    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8h2f\" (UniqueName: \"kubernetes.io/projected/b9055893-3c29-4ff9-b0e6-8c0f9c84117d-kube-api-access-z8h2f\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:44:50.775175 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.775088    2586 generic.go:356] "Generic (PLEG): container finished" podID="b9055893-3c29-4ff9-b0e6-8c0f9c84117d" containerID="70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4" exitCode=0
Sep 17 19:44:50.775175 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.775133    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"
Sep 17 19:44:50.775175 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.775144    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr" event={"ID":"b9055893-3c29-4ff9-b0e6-8c0f9c84117d","Type":"ContainerDied","Data":"70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4"}
Sep 17 19:44:50.775466 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.775186    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr" event={"ID":"b9055893-3c29-4ff9-b0e6-8c0f9c84117d","Type":"ContainerDied","Data":"2a1e5a7ac0e56b8e1eab86e3c428bf3b3ac77a7133bc8ebfb7a939f853f0b30d"}
Sep 17 19:44:50.775466 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.775208    2586 scope.go:117] "RemoveContainer" containerID="70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4"
Sep 17 19:44:50.785684 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.785663    2586 scope.go:117] "RemoveContainer" containerID="70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4"
Sep 17 19:44:50.786013 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:44:50.785990    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4\": container with ID starting with 70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4 not found: ID does not exist" containerID="70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4"
Sep 17 19:44:50.786073 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.786026    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4"} err="failed to get container status \"70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4\": rpc error: code = NotFound desc = could not find container \"70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4\": container with ID starting with 70897060a21bab60bb4f3632d64f67d8063e5dc87508b346987fd913dbbc78c4 not found: ID does not exist"
Sep 17 19:44:50.798254 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.798223    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"]
Sep 17 19:44:50.802142 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:50.802118    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-g8cnr"]
Sep 17 19:44:51.459911 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:44:51.459876    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9055893-3c29-4ff9-b0e6-8c0f9c84117d" path="/var/lib/kubelet/pods/b9055893-3c29-4ff9-b0e6-8c0f9c84117d/volumes"
Sep 17 19:46:52.389811 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.389715    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"]
Sep 17 19:46:52.390437 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.390192    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9055893-3c29-4ff9-b0e6-8c0f9c84117d" containerName="main"
Sep 17 19:46:52.390437 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.390206    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9055893-3c29-4ff9-b0e6-8c0f9c84117d" containerName="main"
Sep 17 19:46:52.390437 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.390278    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9055893-3c29-4ff9-b0e6-8c0f9c84117d" containerName="main"
Sep 17 19:46:52.393368 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.393345    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.396077 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.396057    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 19:46:52.396209 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.396188    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 19:46:52.397330 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.397313    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 19:46:52.397415 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.397349    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-dx5j7\""
Sep 17 19:46:52.404868 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.404825    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"]
Sep 17 19:46:52.521052 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.521011    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9tp\" (UniqueName: \"kubernetes.io/projected/c15a6901-10e2-44d5-a730-5104b80ca413-kube-api-access-df9tp\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.521052 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.521059    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.521289 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.521096    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.521289 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.521163    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.521289 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.521230    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.521289 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.521259    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c15a6901-10e2-44d5-a730-5104b80ca413-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.622222 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.622190    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.622427 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.622237    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.622427 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.622371    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.622427 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.622412    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c15a6901-10e2-44d5-a730-5104b80ca413-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.622609 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.622455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9tp\" (UniqueName: \"kubernetes.io/projected/c15a6901-10e2-44d5-a730-5104b80ca413-kube-api-access-df9tp\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.622609 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.622492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.622722 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.622698    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.622779 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.622736    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.622818 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.622780    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.624685 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.624653    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.625068 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.625048    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c15a6901-10e2-44d5-a730-5104b80ca413-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.633191 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.633167    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9tp\" (UniqueName: \"kubernetes.io/projected/c15a6901-10e2-44d5-a730-5104b80ca413-kube-api-access-df9tp\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bbnxl\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.706081 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.705975    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:52.982669 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:52.982641    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"]
Sep 17 19:46:52.984185 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:46:52.984147    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15a6901_10e2_44d5_a730_5104b80ca413.slice/crio-a230d202c1cabc9261161995dbcb9ca8160514195d940487b4cc79b2851c3c75 WatchSource:0}: Error finding container a230d202c1cabc9261161995dbcb9ca8160514195d940487b4cc79b2851c3c75: Status 404 returned error can't find the container with id a230d202c1cabc9261161995dbcb9ca8160514195d940487b4cc79b2851c3c75
Sep 17 19:46:53.447527 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:53.447479    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl" event={"ID":"c15a6901-10e2-44d5-a730-5104b80ca413","Type":"ContainerStarted","Data":"b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7"}
Sep 17 19:46:53.448058 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:53.447572    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl" event={"ID":"c15a6901-10e2-44d5-a730-5104b80ca413","Type":"ContainerStarted","Data":"a230d202c1cabc9261161995dbcb9ca8160514195d940487b4cc79b2851c3c75"}
Sep 17 19:46:53.471154 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:53.471096    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl" podStartSLOduration=1.471078772 podStartE2EDuration="1.471078772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:46:53.467600432 +0000 UTC m=+4034.327425079" watchObservedRunningTime="2026-09-17 19:46:53.471078772 +0000 UTC m=+4034.330903465"
Sep 17 19:46:53.706523 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:53.706415    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:55.336216 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:55.336168    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"]
Sep 17 19:46:55.551385 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:46:55.551344    2586 secret.go:189] Couldn't get secret e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 17 19:46:55.551567 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:46:55.551428    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c15a6901-10e2-44d5-a730-5104b80ca413-tls-certs podName:c15a6901-10e2-44d5-a730-5104b80ca413 nodeName:}" failed. No retries permitted until 2026-09-17 19:46:56.051408059 +0000 UTC m=+4036.911232703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c15a6901-10e2-44d5-a730-5104b80ca413-tls-certs") pod "rolling-upgrade-test-kserve-6dd8dbf85-bbnxl" (UID: "c15a6901-10e2-44d5-a730-5104b80ca413") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 17 19:46:56.055664 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:46:56.055625    2586 secret.go:189] Couldn't get secret e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 17 19:46:56.055870 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:46:56.055704    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c15a6901-10e2-44d5-a730-5104b80ca413-tls-certs podName:c15a6901-10e2-44d5-a730-5104b80ca413 nodeName:}" failed. No retries permitted until 2026-09-17 19:46:57.055690193 +0000 UTC m=+4037.915514820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c15a6901-10e2-44d5-a730-5104b80ca413-tls-certs") pod "rolling-upgrade-test-kserve-6dd8dbf85-bbnxl" (UID: "c15a6901-10e2-44d5-a730-5104b80ca413") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 17 19:46:56.474283 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:46:56.474244    2586 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:46:56.474283 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:46:56.474283 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:46:56.474283 ip-10-0-129-204 kubenswrapper[2586]:  > containerID="b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7" cmd=["/bin/sleep","15"]
Sep 17 19:46:56.474790 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.474300    2586 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl" message="" err=<
Sep 17 19:46:56.474790 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:46:56.474790 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:46:56.474790 ip-10-0-129-204 kubenswrapper[2586]:  >
Sep 17 19:46:56.474790 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:46:56.474312    2586 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:46:56.474790 ip-10-0-129-204 kubenswrapper[2586]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:46:56.474790 ip-10-0-129-204 kubenswrapper[2586]:         , stdout: , stderr: , exit code -1
Sep 17 19:46:56.474790 ip-10-0-129-204 kubenswrapper[2586]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl" podUID="c15a6901-10e2-44d5-a730-5104b80ca413" containerName="main" containerID="cri-o://b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7"
Sep 17 19:46:56.474790 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.474349    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl" podUID="c15a6901-10e2-44d5-a730-5104b80ca413" containerName="main" containerID="cri-o://b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7" gracePeriod=60
Sep 17 19:46:56.734932 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.734859    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:56.863045 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863007    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-model-cache\") pod \"c15a6901-10e2-44d5-a730-5104b80ca413\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") "
Sep 17 19:46:56.863257 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863075    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c15a6901-10e2-44d5-a730-5104b80ca413-tls-certs\") pod \"c15a6901-10e2-44d5-a730-5104b80ca413\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") "
Sep 17 19:46:56.863257 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863103    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-home\") pod \"c15a6901-10e2-44d5-a730-5104b80ca413\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") "
Sep 17 19:46:56.863257 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863159    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-tmp-dir\") pod \"c15a6901-10e2-44d5-a730-5104b80ca413\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") "
Sep 17 19:46:56.863257 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863183    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df9tp\" (UniqueName: \"kubernetes.io/projected/c15a6901-10e2-44d5-a730-5104b80ca413-kube-api-access-df9tp\") pod \"c15a6901-10e2-44d5-a730-5104b80ca413\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") "
Sep 17 19:46:56.863257 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863227    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-dshm\") pod \"c15a6901-10e2-44d5-a730-5104b80ca413\" (UID: \"c15a6901-10e2-44d5-a730-5104b80ca413\") "
Sep 17 19:46:56.863529 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863340    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-model-cache" (OuterVolumeSpecName: "model-cache") pod "c15a6901-10e2-44d5-a730-5104b80ca413" (UID: "c15a6901-10e2-44d5-a730-5104b80ca413"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:56.863529 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863412    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-home" (OuterVolumeSpecName: "home") pod "c15a6901-10e2-44d5-a730-5104b80ca413" (UID: "c15a6901-10e2-44d5-a730-5104b80ca413"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:56.863529 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863425    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c15a6901-10e2-44d5-a730-5104b80ca413" (UID: "c15a6901-10e2-44d5-a730-5104b80ca413"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:56.863529 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863499    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-home\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:46:56.863725 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863547    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-tmp-dir\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:46:56.863725 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.863565    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-model-cache\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:46:56.865704 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.865680    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-dshm" (OuterVolumeSpecName: "dshm") pod "c15a6901-10e2-44d5-a730-5104b80ca413" (UID: "c15a6901-10e2-44d5-a730-5104b80ca413"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:56.865704 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.865697    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15a6901-10e2-44d5-a730-5104b80ca413-kube-api-access-df9tp" (OuterVolumeSpecName: "kube-api-access-df9tp") pod "c15a6901-10e2-44d5-a730-5104b80ca413" (UID: "c15a6901-10e2-44d5-a730-5104b80ca413"). InnerVolumeSpecName "kube-api-access-df9tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:46:56.865858 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.865761    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15a6901-10e2-44d5-a730-5104b80ca413-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c15a6901-10e2-44d5-a730-5104b80ca413" (UID: "c15a6901-10e2-44d5-a730-5104b80ca413"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:46:56.964264 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.964227    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9tp\" (UniqueName: \"kubernetes.io/projected/c15a6901-10e2-44d5-a730-5104b80ca413-kube-api-access-df9tp\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:46:56.964264 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.964258    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c15a6901-10e2-44d5-a730-5104b80ca413-dshm\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:46:56.964264 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:56.964268    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c15a6901-10e2-44d5-a730-5104b80ca413-tls-certs\") on node \"ip-10-0-129-204.ec2.internal\" DevicePath \"\""
Sep 17 19:46:57.471983 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:57.471955    2586 generic.go:356] "Generic (PLEG): container finished" podID="c15a6901-10e2-44d5-a730-5104b80ca413" containerID="b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7" exitCode=0
Sep 17 19:46:57.472135 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:57.472000    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"
Sep 17 19:46:57.472135 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:57.472025    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl" event={"ID":"c15a6901-10e2-44d5-a730-5104b80ca413","Type":"ContainerDied","Data":"b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7"}
Sep 17 19:46:57.472135 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:57.472087    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl" event={"ID":"c15a6901-10e2-44d5-a730-5104b80ca413","Type":"ContainerDied","Data":"a230d202c1cabc9261161995dbcb9ca8160514195d940487b4cc79b2851c3c75"}
Sep 17 19:46:57.472135 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:57.472128    2586 scope.go:117] "RemoveContainer" containerID="b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7"
Sep 17 19:46:57.483380 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:57.483157    2586 scope.go:117] "RemoveContainer" containerID="b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7"
Sep 17 19:46:57.483676 ip-10-0-129-204 kubenswrapper[2586]: E0917 19:46:57.483526    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7\": container with ID starting with b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7 not found: ID does not exist" containerID="b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7"
Sep 17 19:46:57.483676 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:57.483554    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7"} err="failed to get container status \"b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7\": rpc error: code = NotFound desc = could not find container \"b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7\": container with ID starting with b82f4004fe4826f04888592790b90fa96102d2dd3b3101cfb8e14633c6c20af7 not found: ID does not exist"
Sep 17 19:46:57.491265 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:57.491238    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"]
Sep 17 19:46:57.493290 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:57.493268    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bbnxl"]
Sep 17 19:46:59.458398 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:46:59.458360    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c15a6901-10e2-44d5-a730-5104b80ca413" path="/var/lib/kubelet/pods/c15a6901-10e2-44d5-a730-5104b80ca413/volumes"
Sep 17 19:48:08.892562 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:08.892530    2586 ???:1] "http: TLS handshake error from 10.0.140.194:58892: EOF"
Sep 17 19:48:08.961264 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:08.961226    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-f4n7q_4dc816e7-abf9-4437-b226-ec80b9378fec/discovery/0.log"
Sep 17 19:48:09.798760 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:09.798714    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-f4n7q_4dc816e7-abf9-4437-b226-ec80b9378fec/discovery/0.log"
Sep 17 19:48:10.815339 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:10.815302    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-bhdd9_f30db5da-b425-4dd7-8ceb-18e9b10ce350/manager/0.log"
Sep 17 19:48:10.829578 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:10.829547    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-h85px_0bf2898d-6df1-4592-a611-d0d420ff70c6/manager/0.log"
Sep 17 19:48:10.845210 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:10.845186    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-5sqq4_a5b293af-3321-46ca-8444-b93ed864838c/kuadrant-console-plugin/0.log"
Sep 17 19:48:11.022025 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:11.021989    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-2nph5_cbb4a599-0b6e-450d-903b-f7f5b8780f17/manager/0.log"
Sep 17 19:48:16.778426 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.778388    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l7vd6/must-gather-gwv2r"]
Sep 17 19:48:16.778892 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.778784    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15a6901-10e2-44d5-a730-5104b80ca413" containerName="main"
Sep 17 19:48:16.778892 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.778797    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15a6901-10e2-44d5-a730-5104b80ca413" containerName="main"
Sep 17 19:48:16.778892 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.778883    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c15a6901-10e2-44d5-a730-5104b80ca413" containerName="main"
Sep 17 19:48:16.782196 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.782174    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7vd6/must-gather-gwv2r"
Sep 17 19:48:16.784959 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.784937    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7vd6\"/\"kube-root-ca.crt\""
Sep 17 19:48:16.786062 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.786034    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l7vd6\"/\"default-dockercfg-z24dx\""
Sep 17 19:48:16.786062 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.786050    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7vd6\"/\"openshift-service-ca.crt\""
Sep 17 19:48:16.791028 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.790999    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7vd6/must-gather-gwv2r"]
Sep 17 19:48:16.846063 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.846021    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/25f19416-a147-424a-b466-4484d0112bc1-must-gather-output\") pod \"must-gather-gwv2r\" (UID: \"25f19416-a147-424a-b466-4484d0112bc1\") " pod="openshift-must-gather-l7vd6/must-gather-gwv2r"
Sep 17 19:48:16.846266 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.846081    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g25bt\" (UniqueName: \"kubernetes.io/projected/25f19416-a147-424a-b466-4484d0112bc1-kube-api-access-g25bt\") pod \"must-gather-gwv2r\" (UID: \"25f19416-a147-424a-b466-4484d0112bc1\") " pod="openshift-must-gather-l7vd6/must-gather-gwv2r"
Sep 17 19:48:16.947275 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.947233    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/25f19416-a147-424a-b466-4484d0112bc1-must-gather-output\") pod \"must-gather-gwv2r\" (UID: \"25f19416-a147-424a-b466-4484d0112bc1\") " pod="openshift-must-gather-l7vd6/must-gather-gwv2r"
Sep 17 19:48:16.947467 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.947310    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g25bt\" (UniqueName: \"kubernetes.io/projected/25f19416-a147-424a-b466-4484d0112bc1-kube-api-access-g25bt\") pod \"must-gather-gwv2r\" (UID: \"25f19416-a147-424a-b466-4484d0112bc1\") " pod="openshift-must-gather-l7vd6/must-gather-gwv2r"
Sep 17 19:48:16.947636 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.947613    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/25f19416-a147-424a-b466-4484d0112bc1-must-gather-output\") pod \"must-gather-gwv2r\" (UID: \"25f19416-a147-424a-b466-4484d0112bc1\") " pod="openshift-must-gather-l7vd6/must-gather-gwv2r"
Sep 17 19:48:16.962591 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:16.962559    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g25bt\" (UniqueName: \"kubernetes.io/projected/25f19416-a147-424a-b466-4484d0112bc1-kube-api-access-g25bt\") pod \"must-gather-gwv2r\" (UID: \"25f19416-a147-424a-b466-4484d0112bc1\") " pod="openshift-must-gather-l7vd6/must-gather-gwv2r"
Sep 17 19:48:17.092732 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:17.092693    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7vd6/must-gather-gwv2r"
Sep 17 19:48:17.383151 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:17.383068    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7vd6/must-gather-gwv2r"]
Sep 17 19:48:17.386205 ip-10-0-129-204 kubenswrapper[2586]: W0917 19:48:17.386174    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25f19416_a147_424a_b466_4484d0112bc1.slice/crio-e3e05ee10b5335e9c1beaeba9ac0b4b887c1166f8056eb192c914d833efae0e1 WatchSource:0}: Error finding container e3e05ee10b5335e9c1beaeba9ac0b4b887c1166f8056eb192c914d833efae0e1: Status 404 returned error can't find the container with id e3e05ee10b5335e9c1beaeba9ac0b4b887c1166f8056eb192c914d833efae0e1
Sep 17 19:48:17.388231 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:17.388212    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:48:17.930214 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:17.930173    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7vd6/must-gather-gwv2r" event={"ID":"25f19416-a147-424a-b466-4484d0112bc1","Type":"ContainerStarted","Data":"e3e05ee10b5335e9c1beaeba9ac0b4b887c1166f8056eb192c914d833efae0e1"}
Sep 17 19:48:18.941209 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:18.941152    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7vd6/must-gather-gwv2r" event={"ID":"25f19416-a147-424a-b466-4484d0112bc1","Type":"ContainerStarted","Data":"85e18e0a12341f4fb4865aad04f4043b48bd51c447e0fabddf2d6bd06760470e"}
Sep 17 19:48:18.941738 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:18.941225    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7vd6/must-gather-gwv2r" event={"ID":"25f19416-a147-424a-b466-4484d0112bc1","Type":"ContainerStarted","Data":"ca18a308b880df80bd8df4e7631616762e08218074c0ca6c3d17c4b0ee212b9b"}
Sep 17 19:48:18.958553 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:18.958493    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l7vd6/must-gather-gwv2r" podStartSLOduration=2.178000926 podStartE2EDuration="2.958471162s" totalImagesPullingTime="780.470236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:48:18.957717354 +0000 UTC m=+4119.817542014" watchObservedRunningTime="2026-09-17 19:48:18.958471162 +0000 UTC m=+4119.818295801"
Sep 17 19:48:19.793874 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:19.793820    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xnrlh_670d8bfc-6813-421f-941d-c7e35a57f000/global-pull-secret-syncer/0.log"
Sep 17 19:48:19.936540 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:19.936506    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xwwg7_1463c73f-f522-4ea3-86f1-4ce25079b61c/konnectivity-agent/0.log"
Sep 17 19:48:19.960110 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:19.960078    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-204.ec2.internal_894af7d1d57be17aaf3ffac63d929c3b/haproxy/0.log"
Sep 17 19:48:24.653722 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:24.653601    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-bhdd9_f30db5da-b425-4dd7-8ceb-18e9b10ce350/manager/0.log"
Sep 17 19:48:24.692752 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:24.692723    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-h85px_0bf2898d-6df1-4592-a611-d0d420ff70c6/manager/0.log"
Sep 17 19:48:24.729000 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:24.728953    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-5sqq4_a5b293af-3321-46ca-8444-b93ed864838c/kuadrant-console-plugin/0.log"
Sep 17 19:48:24.942626 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:24.942540    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-2nph5_cbb4a599-0b6e-450d-903b-f7f5b8780f17/manager/0.log"
Sep 17 19:48:26.505165 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:26.505052    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-b5vzc_0c0ac57d-1215-47ce-88ab-f82dd383dc4f/kube-state-metrics/0.log"
Sep 17 19:48:26.528277 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:26.528229    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-b5vzc_0c0ac57d-1215-47ce-88ab-f82dd383dc4f/kube-rbac-proxy-main/0.log"
Sep 17 19:48:26.551721 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:26.551692    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-b5vzc_0c0ac57d-1215-47ce-88ab-f82dd383dc4f/kube-rbac-proxy-self/0.log"
Sep 17 19:48:26.727949 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:26.727901    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzlvr_df5b11cf-b032-4878-b86d-af0b9731236a/node-exporter/0.log"
Sep 17 19:48:26.754858 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:26.754764    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzlvr_df5b11cf-b032-4878-b86d-af0b9731236a/kube-rbac-proxy/0.log"
Sep 17 19:48:26.784777 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:26.784706    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzlvr_df5b11cf-b032-4878-b86d-af0b9731236a/init-textfile/0.log"
Sep 17 19:48:26.904652 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:26.904624    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-6twll_6f7a9e79-81ac-4cfd-a067-33df41345c65/kube-rbac-proxy-main/0.log"
Sep 17 19:48:26.930631 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:26.930604    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-6twll_6f7a9e79-81ac-4cfd-a067-33df41345c65/kube-rbac-proxy-self/0.log"
Sep 17 19:48:26.957479 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:26.957437    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-6twll_6f7a9e79-81ac-4cfd-a067-33df41345c65/openshift-state-metrics/0.log"
Sep 17 19:48:27.233278 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:27.233240    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-59x8f_d3997c3c-be8b-49dc-aeda-8e39adfd65e8/prometheus-operator-admission-webhook/0.log"
Sep 17 19:48:27.262697 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:27.262652    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d8b9bb886-jz9dn_9030b38c-0472-4cd4-bc5d-65371412f054/telemeter-client/0.log"
Sep 17 19:48:27.286663 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:27.286606    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d8b9bb886-jz9dn_9030b38c-0472-4cd4-bc5d-65371412f054/reload/0.log"
Sep 17 19:48:27.311893 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:27.311860    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d8b9bb886-jz9dn_9030b38c-0472-4cd4-bc5d-65371412f054/kube-rbac-proxy/0.log"
Sep 17 19:48:27.347109 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:27.347081    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54d8d5b55c-pddhs_e3855f0a-c397-410b-a52c-69cef688a2b1/thanos-query/0.log"
Sep 17 19:48:27.369702 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:27.369673    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54d8d5b55c-pddhs_e3855f0a-c397-410b-a52c-69cef688a2b1/kube-rbac-proxy-web/0.log"
Sep 17 19:48:27.391433 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:27.391405    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54d8d5b55c-pddhs_e3855f0a-c397-410b-a52c-69cef688a2b1/kube-rbac-proxy/0.log"
Sep 17 19:48:27.412588 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:27.412557    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54d8d5b55c-pddhs_e3855f0a-c397-410b-a52c-69cef688a2b1/prom-label-proxy/0.log"
Sep 17 19:48:27.435755 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:27.435711    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54d8d5b55c-pddhs_e3855f0a-c397-410b-a52c-69cef688a2b1/kube-rbac-proxy-rules/0.log"
Sep 17 19:48:27.457560 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:27.457520    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54d8d5b55c-pddhs_e3855f0a-c397-410b-a52c-69cef688a2b1/kube-rbac-proxy-metrics/0.log"
Sep 17 19:48:30.009733 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.009703    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-fc976d4b7-m2j4h_8e923894-d919-4eaf-a313-0643ba059228/console/0.log"
Sep 17 19:48:30.430165 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.430126    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"]
Sep 17 19:48:30.433854 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.433815    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.444214 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.444184    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"]
Sep 17 19:48:30.493512 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.493470    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-proc\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.493811 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.493789    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-lib-modules\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.494076 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.494044    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-podres\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.494223 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.494209    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-sys\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.494333 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.494320    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9xv\" (UniqueName: \"kubernetes.io/projected/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-kube-api-access-cl9xv\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.595634 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.595585    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-podres\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.595854 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.595656    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-sys\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.595854 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.595691    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9xv\" (UniqueName: \"kubernetes.io/projected/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-kube-api-access-cl9xv\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.596046 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.595873    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-proc\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.596046 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.595899    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-lib-modules\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.596155 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.596068    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-lib-modules\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.596208 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.596160    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-podres\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.596208 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.596200    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-sys\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.597475 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.597448    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-proc\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.606611 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.606552    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9xv\" (UniqueName: \"kubernetes.io/projected/34911e8c-10e6-430b-8bee-3b8f7e8aaf8b-kube-api-access-cl9xv\") pod \"perf-node-gather-daemonset-q85xd\" (UID: \"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:30.748938 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:30.748810    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:31.050962 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:31.050866    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"]
Sep 17 19:48:31.842297 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:31.842267    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tx5nx_d9bf4517-1532-40ee-b569-0aed944e652b/dns/0.log"
Sep 17 19:48:31.863759 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:31.863730    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tx5nx_d9bf4517-1532-40ee-b569-0aed944e652b/kube-rbac-proxy/0.log"
Sep 17 19:48:31.889204 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:31.889176    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dxrln_6caa7aaf-fa9b-482d-b166-30b9f3515174/dns-node-resolver/0.log"
Sep 17 19:48:32.053188 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:32.053151    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd" event={"ID":"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b","Type":"ContainerStarted","Data":"af649e4306938eedf8303ac466ce40b212e92ed8c6657413f9e0f1e79559afe8"}
Sep 17 19:48:32.053589 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:32.053205    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd" event={"ID":"34911e8c-10e6-430b-8bee-3b8f7e8aaf8b","Type":"ContainerStarted","Data":"2deb40d2891a398df201376216a7c9546d73297fde35893306f4ed2caef6aa5b"}
Sep 17 19:48:32.053589 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:32.053259    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:32.070096 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:32.070037    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd" podStartSLOduration=2.070019305 podStartE2EDuration="2.070019305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:48:32.068410425 +0000 UTC m=+4132.928235089" watchObservedRunningTime="2026-09-17 19:48:32.070019305 +0000 UTC m=+4132.929843953"
Sep 17 19:48:32.531602 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:32.531570    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8mvc8_1af79fd6-6d59-4d91-bbee-226e76b02e32/node-ca/0.log"
Sep 17 19:48:33.554607 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:33.554570    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-f4n7q_4dc816e7-abf9-4437-b226-ec80b9378fec/discovery/0.log"
Sep 17 19:48:34.144292 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:34.144251    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cx55j_584cdffd-f790-4665-acc8-f7cdd2d1f46f/serve-healthcheck-canary/0.log"
Sep 17 19:48:34.840385 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:34.840352    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-66znr_81b4a5e2-4e3d-416f-b42b-3e43ceb3365c/kube-rbac-proxy/0.log"
Sep 17 19:48:34.874093 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:34.874065    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-66znr_81b4a5e2-4e3d-416f-b42b-3e43ceb3365c/exporter/0.log"
Sep 17 19:48:34.902059 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:34.902027    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-66znr_81b4a5e2-4e3d-416f-b42b-3e43ceb3365c/extractor/0.log"
Sep 17 19:48:38.074355 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:38.074318    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-q85xd"
Sep 17 19:48:38.074928 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:38.074518    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-77dw8_de36c1d9-f8cb-4780-8a46-410602b7c54c/openshift-lws-operator/0.log"
Sep 17 19:48:38.788291 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:38.788255    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7dcf484667-bf8ww_2acd3526-1681-4bee-b51f-7f51a7562721/manager/0.log"
Sep 17 19:48:39.264826 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:39.264801    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-wnh9c_34a7e137-a8dd-4f9e-b417-7cf6a0cfaa48/server/0.log"
Sep 17 19:48:40.038748 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:40.038705    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-2n5nn_4bb34e51-7e61-4c03-b670-1c34b495372c/manager/0.log"
Sep 17 19:48:40.121906 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:40.121189    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-k24t2_dda9b6ff-71bf-4dcc-9050-513c47e80e8a/seaweedfs/0.log"
Sep 17 19:48:40.340671 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:40.340590    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-f4n7q_4dc816e7-abf9-4437-b226-ec80b9378fec/discovery/0.log"
Sep 17 19:48:41.521894 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:41.521855    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-f4n7q_4dc816e7-abf9-4437-b226-ec80b9378fec/discovery/0.log"
Sep 17 19:48:46.443970 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:46.443934    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5lkxr_fb7d17e7-5508-41b4-83d5-e5130b662b81/kube-multus/0.log"
Sep 17 19:48:46.465066 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:46.465033    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2bgpb_228d79c5-beb7-4a41-b877-c68fae3efc15/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:48:46.485460 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:46.485433    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2bgpb_228d79c5-beb7-4a41-b877-c68fae3efc15/egress-router-binary-copy/0.log"
Sep 17 19:48:46.511661 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:46.511628    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2bgpb_228d79c5-beb7-4a41-b877-c68fae3efc15/cni-plugins/0.log"
Sep 17 19:48:46.531751 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:46.531722    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2bgpb_228d79c5-beb7-4a41-b877-c68fae3efc15/bond-cni-plugin/0.log"
Sep 17 19:48:46.552628 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:46.552597    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2bgpb_228d79c5-beb7-4a41-b877-c68fae3efc15/routeoverride-cni/0.log"
Sep 17 19:48:46.572996 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:46.572967    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2bgpb_228d79c5-beb7-4a41-b877-c68fae3efc15/whereabouts-cni-bincopy/0.log"
Sep 17 19:48:46.593253 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:46.593226    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2bgpb_228d79c5-beb7-4a41-b877-c68fae3efc15/whereabouts-cni/0.log"
Sep 17 19:48:47.136818 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:47.136783    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vhljw_e8969506-84fa-4bcc-a5f6-383cbe2b041f/network-metrics-daemon/0.log"
Sep 17 19:48:47.167403 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:47.167373    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vhljw_e8969506-84fa-4bcc-a5f6-383cbe2b041f/kube-rbac-proxy/0.log"
Sep 17 19:48:48.228728 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:48.228679    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-controller/0.log"
Sep 17 19:48:48.245850 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:48.245790    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/0.log"
Sep 17 19:48:48.246027 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:48.245857    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovn-acl-logging/1.log"
Sep 17 19:48:48.263055 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:48.263010    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/kube-rbac-proxy-node/0.log"
Sep 17 19:48:48.283881 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:48.283851    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:48:48.304878 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:48.304827    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/northd/0.log"
Sep 17 19:48:48.326371 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:48.326338    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/nbdb/0.log"
Sep 17 19:48:48.347297 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:48.347271    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/sbdb/0.log"
Sep 17 19:48:48.419143 ip-10-0-129-204 kubenswrapper[2586]: I0917 19:48:48.419108    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjczz_f1cf4c16-7f5e-415b-848a-fef07ac8160b/ovnkube-controller/0.log"