Sep 17 15:38:37.987185 ip-10-0-135-243 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:38:38.407167 ip-10-0-135-243 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:38.407167 ip-10-0-135-243 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:38:38.407167 ip-10-0-135-243 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:38.407167 ip-10-0-135-243 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:38:38.407167 ip-10-0-135-243 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:38.408280 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.408183    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:38:38.412718 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412697    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:38.412718 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412718    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412722    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412726    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412729    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412732    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412734    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412737    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412740    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412745    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412749    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412752    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412756    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412759    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412766    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412769    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412772    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412774    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412778    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412780    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412783    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:38.412800 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412785    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412788    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412791    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412794    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412796    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412799    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412802    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412805    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412808    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412811    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412814    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412817    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412831    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412834    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412838    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412841    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412843    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412846    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412849    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412852    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:38.413290 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412855    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412857    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412860    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412863    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412866    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412869    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412872    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412876    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412880    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412882    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412885    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412888    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412891    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412894    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412898    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412901    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412903    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412906    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412909    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:38.413791 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412912    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412914    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412917    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412919    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412922    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412924    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412928    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412932    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412936    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412939    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412942    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412945    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412947    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412950    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412952    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412956    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412959    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412962    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412965    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412968    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:38.414275 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412970    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412973    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412976    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412978    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412981    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412984    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412987    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412990    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412992    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412995    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.412998    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413000    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413003    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413005    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413008    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413011    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413013    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413016    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413018    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413730    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:38.414785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413736    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413739    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413742    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413745    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413749    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413751    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413754    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413757    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413762    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413767    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413769    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413772    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413775    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413778    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413781    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413784    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413787    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413790    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413793    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:38.415398 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413798    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413802    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413805    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413807    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413810    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413813    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413816    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413819    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413840    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413844    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413848    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413852    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413858    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413861    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413863    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413866    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413869    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413872    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413875    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:38.415909 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413879    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413883    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413888    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413891    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413894    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413897    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413899    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413904    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413906    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413909    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413912    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413915    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413917    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413920    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413923    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413925    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413928    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413930    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413933    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413935    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:38.416374 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413938    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413941    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413943    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413947    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413949    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413952    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413955    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413957    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413960    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413963    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413965    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413968    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413970    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413973    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413975    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413979    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413986    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413989    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413992    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413994    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:38.416897 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413997    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.413999    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414002    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414005    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414007    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414010    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414012    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414015    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414017    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414020    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414022    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414025    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414027    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414030    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414033    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414035    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414037    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414040    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414043    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414045    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:38.417436 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414476    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414483    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414486    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414490    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414494    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414497    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414500    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414503    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414506    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414509    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414512    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414515    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414517    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414520    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414523    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414526    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414529    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414532    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414534    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414538    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:38.417983 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414540    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414543    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414545    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414548    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414551    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414553    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414556    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414560    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414564    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414567    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414571    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414574    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414577    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414579    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414582    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414585    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414588    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414591    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414594    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:38.418474 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414596    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414600    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414604    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414606    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414609    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414612    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414614    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414617    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414619    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414622    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414624    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414627    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414630    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414632    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414635    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414637    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414640    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414642    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414645    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414648    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:38.418971 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414651    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414654    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414656    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414659    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414662    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414665    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414667    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414670    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414673    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414676    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414679    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414681    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414684    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414686    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414689    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414695    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414698    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414700    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414704    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:38.419494 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414708    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414711    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414714    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414716    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414719    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414722    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414724    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414727    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414729    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414732    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414734    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414737    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414739    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414742    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414744    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414747    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414752    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414755    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414758    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414760    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:38.419988 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.414763    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414857    2555 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414893    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414907    2555 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414912    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414916    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414919    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414924    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414929    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414938    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414941    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414945    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414948    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414951    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414954    2555 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414957    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414961    2555 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414964    2555 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414967    2555 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414975    2555 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414978    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414981    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414984    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414987    2555 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:38:38.420484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414991    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414995    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.414998    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415002    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415006    2555 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415009    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415012    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415015    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415018    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415023    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415026    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415029    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415032    2555 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415035    2555 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415038    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415045    2555 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415048    2555 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415051    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415060    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415063    2555 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415067    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415070    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415073    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415076    2555 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415079    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:38:38.421131 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415082    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415085    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415089    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415093    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415096    2555 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415099    2555 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415103    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415107    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415110    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415113    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415117    2555 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415121    2555 flags.go:64] FLAG: --help="false"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415124    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415127    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415130    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415133    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415137    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415140    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415143    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415146    2555 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415149    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415151    2555 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415155    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415158    2555 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:38:38.421768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415161    2555 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415164    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415173    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415176    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415179    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415182    2555 flags.go:64] FLAG: --lock-file=""
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415185    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415188    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415191    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415197    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415199    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415202    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415205    2555 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415208    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415211    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415214    2555 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415217    2555 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415221    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415228    2555 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415233    2555 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415236    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415239    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415242    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415245    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415248    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:38:38.422643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415251    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415254    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415262    2555 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415265    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415268    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415271    2555 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415274    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415280    2555 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415283    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415286    2555 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415289    2555 flags.go:64] FLAG: --port="10250"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415293    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415296    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-092c30e0afb30f86a"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415299    2555 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415302    2555 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415305    2555 flags.go:64] FLAG: --register-node="true"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415308    2555 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415312    2555 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415315    2555 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415318    2555 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415321    2555 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415325    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415328    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415331    2555 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415334    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:38:38.423726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415337    2555 flags.go:64] FLAG: --runonce="false"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415342    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415345    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415349    2555 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415352    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415355    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415358    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415361    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415364    2555 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415367    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415370    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415373    2555 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415376    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415379    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415383    2555 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415386    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415391    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415394    2555 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415400    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415406    2555 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415409    2555 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415412    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415415    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415418    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415422    2555 flags.go:64] FLAG: --v="2"
Sep 17 15:38:38.424691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415426    2555 flags.go:64] FLAG: --version="false"
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415431    2555 flags.go:64] FLAG: --vmodule=""
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415434    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.415438    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415573    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415576    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415579    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415585    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415589    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415593    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415595    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415599    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415602    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415605    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415608    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415610    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415613    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415616    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415619    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415621    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415624    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:38.425489 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415627    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415629    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415632    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415635    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415638    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415641    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415644    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415647    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415651    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415655    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415657    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415660    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415663    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415666    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415668    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415671    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415673    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415676    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415680    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:38.426449 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415686    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415689    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415693    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415697    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415699    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415703    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415705    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415708    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415711    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415714    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415717    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415719    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415722    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415725    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415727    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415730    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415732    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415735    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415738    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:38.427129 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415743    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415746    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415749    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415752    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415756    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415759    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415761    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415764    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415767    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415769    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415772    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415775    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415778    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415783    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415785    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415788    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415791    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415794    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415797    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:38.427602 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415799    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415802    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415805    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415807    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415810    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415813    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415816    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415818    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415836    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415839    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415842    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415844    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415847    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415850    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415853    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415856    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415859    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415861    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415864    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415867    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:38.428108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415869    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415872    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415874    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415877    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.415880    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.416664    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.424420    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.424445    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.425353    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.425372    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425469    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425479    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425485    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425490    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425494    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:38.428604 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425498    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425502    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425507    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425511    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425514    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425518    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425523    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425527    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425531    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425536    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425540    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425544    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425548    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425553    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425557    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425561    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425565    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425569    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425573    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425577    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425582    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:38.429076 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425586    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425591    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425595    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425602    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425607    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425611    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425616    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425623    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425629    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425634    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425639    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425643    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425647    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425654    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425659    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425664    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425668    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425672    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425676    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:38.429636 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425680    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425685    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425689    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425693    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425697    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425701    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425706    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425712    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425716    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425721    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425725    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425729    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425733    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425738    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425743    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425747    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425751    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425756    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425761    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425765    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:38.430141 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425769    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425773    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425777    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425781    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425785    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425791    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425795    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425799    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425803    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425807    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425811    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425815    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425835    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425840    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425845    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425849    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425854    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425858    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425862    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:38.430647 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425867    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425871    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425876    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425880    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425884    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425888    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425893    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425896    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425901    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425906    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425910    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425915    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425920    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425924    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:38.431134 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.425930    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.425939    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426132    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426141    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426146    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426150    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426154    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426160    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426164    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426168    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426172    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426176    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426181    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426186    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426190    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:38.431493 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426194    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426198    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426203    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426208    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426213    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426217    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426221    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426228    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426233    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426239    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426244    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426248    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426252    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426256    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426261    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426267    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426271    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426275    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426280    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:38.431883 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426284    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426288    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426292    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426296    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426301    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426305    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426309    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426315    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426320    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426324    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426329    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426333    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426337    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426341    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426345    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426349    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426353    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426358    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426372    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:38.432372 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426377    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426382    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426387    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426391    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426395    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426399    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426404    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426408    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426412    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426416    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426423    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426428    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426432    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426437    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426441    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426445    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426449    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426453    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426458    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426461    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:38.432853 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426465    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426469    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426473    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426478    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426482    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426486    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426491    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426495    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426499    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426503    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426508    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426512    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426517    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426521    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426525    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426530    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426534    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426539    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426544    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426548    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:38.433352 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426552    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426558    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426564    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426571    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426575    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426580    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426584    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:38.426588    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.426597    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.426801    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.428960    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.429943    2555 server.go:1019] "Starting client certificate rotation"
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.430051    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:38:38.433882 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.430093    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:38:38.434271 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.432285    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:38:38.434271 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.433859    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:38:38.445978 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.445954    2555 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:38:38.460843 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.460800    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:38:38.513562 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.513521    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2sr74"
Sep 17 15:38:38.519650 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.519631    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2sr74"
Sep 17 15:38:38.539694 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.539671    2555 log.go:25] "Validated CRI v1 image API"
Sep 17 15:38:38.540698 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.540676    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:38:38.543372 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.543341    2555 fs.go:135] Filesystem UUIDs: map[5de6b10f-ab99-4271-82d2-8d47c2bb839a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9fa560d2-65a4-48fe-beea-3ef47d25dc5e:/dev/nvme0n1p3]
Sep 17 15:38:38.543471 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.543368    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:38:38.548646 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.548518    2555 manager.go:216] Machine: {Timestamp:2026-09-17 15:38:38.547243283 +0000 UTC m=+0.438103818 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100201 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28f60c0323ea30348458d6cc6bacc7 SystemUUID:ec28f60c-0323-ea30-3484-58d6cc6bacc7 BootID:f16d366d-d598-461e-b5b3-0baaa2ddc10a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:c4:69:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:c4:69:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:7f:b6:9d:47:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:38:38.548646 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.548638    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:38:38.548758 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.548741    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:38:38.552142 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.552109    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:38:38.552314 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.552142    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-243.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:38:38.553524 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.553506    2555 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:38:38.553524 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.553520    2555 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:38:38.553667 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.553529    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:38:38.553667 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.553544    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:38:38.554424 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.554407    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:38:38.558151 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.558132    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:38:38.558282 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.558270    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:38:38.558330 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.558306    2555 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:38:38.558330 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.558320    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:38:38.558421 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.558342    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:38:38.558421 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.558355    2555 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:38:38.558421 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.558380    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:38:38.558800 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.558788    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:38:38.558862 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.558808    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:38:38.561576 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.561555    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 15:38:38.563899 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.563882    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:38:38.563971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.563914    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:38:38.565533 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565516    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:38:38.565533 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565537    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:38:38.565668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565543    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:38:38.565668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565548    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:38:38.565668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565569    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:38:38.565668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565578    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:38:38.565668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565597    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:38:38.565668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565603    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:38:38.565668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565609    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:38:38.565668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565615    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:38:38.565668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565631    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:38:38.565668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.565637    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:38:38.566482 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.566469    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:38:38.566482 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.566479    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:38:38.570760 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.570743    2555 server.go:1268] "Started kubelet"
Sep 17 15:38:38.570874 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.570797    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:38:38.571049 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.570956    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:38:38.571168 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.571148    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:38:38.571766 ip-10-0-135-243 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:38:38.575264 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.575240    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:38:38.576653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.576625    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:38.578493 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.578473    2555 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:38:38.578625 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:38.578606    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:38:38.579442 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.579425    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:38.579886 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.579871    2555 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-243.ec2.internal" not found
Sep 17 15:38:38.581815 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.581796    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:38:38.582415 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.582400    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:38:38.582638 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:38.582521    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-243.ec2.internal\" not found"
Sep 17 15:38:38.583653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583030    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:38:38.583653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583032    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:38:38.583653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583054    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:38:38.583653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583176    2555 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:38:38.583653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583184    2555 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:38:38.583653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583274    2555 factory.go:55] Registering systemd factory
Sep 17 15:38:38.583653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583289    2555 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:38:38.583653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583566    2555 factory.go:153] Registering CRI-O factory
Sep 17 15:38:38.583653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583584    2555 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:38:38.583653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583645    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:38:38.584206 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583667    2555 factory.go:103] Registering Raw factory
Sep 17 15:38:38.584206 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.583684    2555 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:38:38.584487 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.584473    2555 manager.go:349] Starting recovery of all containers
Sep 17 15:38:38.585653 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.585628    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:38.590177 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:38.589975    2555 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-243.ec2.internal\" not found" node="ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.595592 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.595575    2555 manager.go:354] Recovery completed
Sep 17 15:38:38.597696 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.597682    2555 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-243.ec2.internal" not found
Sep 17 15:38:38.601967 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.601946    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:38.603853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.603818    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-243.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:38.603946 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.603870    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-243.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:38.603946 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.603885    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-243.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:38.604524 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.604507    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:38:38.604573 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.604524    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:38:38.604573 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.604541    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:38:38.604639 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.604511    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:38.605345 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.605330    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-243.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:38.605398 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.605356    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-243.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:38.605398 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.605367    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-243.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:38.605739 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.605726    2555 policy_none.go:49] "None policy: Start"
Sep 17 15:38:38.605777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.605743    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:38:38.605777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.605754    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:38:38.606902 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.606891    2555 policy_none.go:47] "Start"
Sep 17 15:38:38.641531 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.641514    2555 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:38:38.651330 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:38.641554    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:38:38.651330 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.641568    2555 server.go:87] "Starting device plugin registration server"
Sep 17 15:38:38.651330 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.642033    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:38:38.651330 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.642047    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:38:38.651330 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.642142    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:38:38.651330 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.642225    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:38:38.651330 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.642234    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:38:38.651330 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:38.642898    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:38:38.651330 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:38.642942    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-243.ec2.internal\" not found"
Sep 17 15:38:38.658616 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.658529    2555 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-243.ec2.internal" not found
Sep 17 15:38:38.686923 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.686886    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:38:38.688074 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.688054    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:38:38.688166 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.688121    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:38:38.688166 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.688158    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:38:38.688261 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:38.688208    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:38:38.691222 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.691200    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:38.742611 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.742578    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:38.743640 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.743620    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-243.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:38.743740 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.743654    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-243.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:38.743740 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.743665    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-243.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:38.743740 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.743693    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.755376 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.755347    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.755472 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:38.755383    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-243.ec2.internal\": node \"ip-10-0-135-243.ec2.internal\" not found"
Sep 17 15:38:38.788906 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.788867    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal"]
Sep 17 15:38:38.792017 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.791998    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.793868 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.793852    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.821801 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.821780    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.825381 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.825361    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.874086 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.874049    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:38:38.874086 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.874083    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:38:38.884063 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.884026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eff4a4436cf19c431e9ef9db87e16884-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal\" (UID: \"eff4a4436cf19c431e9ef9db87e16884\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.884063 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.884063    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eff4a4436cf19c431e9ef9db87e16884-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal\" (UID: \"eff4a4436cf19c431e9ef9db87e16884\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.884283 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.884082    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8ce232a731178d99d36cb8d051eb3270-config\") pod \"kube-apiserver-proxy-ip-10-0-135-243.ec2.internal\" (UID: \"8ce232a731178d99d36cb8d051eb3270\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.984885 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.984770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eff4a4436cf19c431e9ef9db87e16884-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal\" (UID: \"eff4a4436cf19c431e9ef9db87e16884\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.984885 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.984806    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8ce232a731178d99d36cb8d051eb3270-config\") pod \"kube-apiserver-proxy-ip-10-0-135-243.ec2.internal\" (UID: \"8ce232a731178d99d36cb8d051eb3270\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.984885 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.984843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eff4a4436cf19c431e9ef9db87e16884-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal\" (UID: \"eff4a4436cf19c431e9ef9db87e16884\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.985128 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.984904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8ce232a731178d99d36cb8d051eb3270-config\") pod \"kube-apiserver-proxy-ip-10-0-135-243.ec2.internal\" (UID: \"8ce232a731178d99d36cb8d051eb3270\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.985128 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.984919    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eff4a4436cf19c431e9ef9db87e16884-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal\" (UID: \"eff4a4436cf19c431e9ef9db87e16884\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:38.985128 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:38.984967    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eff4a4436cf19c431e9ef9db87e16884-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal\" (UID: \"eff4a4436cf19c431e9ef9db87e16884\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:39.176896 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.176859    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:39.177946 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.177917    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal"
Sep 17 15:38:39.429636 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.429539    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:38:39.430274 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.429695    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:38:39.430274 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.429698    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:38:39.430274 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.429695    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:38:39.542534 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.542435    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:33:38 +0000 UTC" deadline="2028-02-20 03:41:46.377545074 +0000 UTC"
Sep 17 15:38:39.542534 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.542525    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12492h3m6.835024425s"
Sep 17 15:38:39.558577 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.558547    2555 apiserver.go:52] "Watching apiserver"
Sep 17 15:38:39.569746 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.569711    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:38:39.570140 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.570119    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-h7dwx","kube-system/konnectivity-agent-p65h2","kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss","openshift-cluster-node-tuning-operator/tuned-phv6p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal","openshift-multus/multus-gxshr","openshift-multus/network-metrics-daemon-b2879","openshift-network-diagnostics/network-check-target-6nj4l","kube-system/global-pull-secret-syncer-nlf6d","openshift-dns/node-resolver-rjjjt","openshift-image-registry/node-ca-4k758","openshift-multus/multus-additional-cni-plugins-pjwg8","openshift-network-operator/iptables-alerter-vqm4m"]
Sep 17 15:38:39.572532 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.572514    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:39.572645 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.572619    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:39.574565 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.574543    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.576496 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.576470    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.578200 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.578179    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-74mqt\""
Sep 17 15:38:39.578340 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.578328    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:38:39.578435 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.578413    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.578968 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.578951    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:38:39.579732 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.579720    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:38:39.579869 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.579852    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:38:39.580377 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.580363    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.580808 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.580790    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:38:39.580915 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.580902    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qt7tp\""
Sep 17 15:38:39.580962 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.580904    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:38:39.581916 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.581887    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:38:39.581916 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.581908    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:38:39.581916 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.581913    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-brk72\""
Sep 17 15:38:39.582102 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.581889    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:38:39.582375 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.582352    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:39.582458 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.582435    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:39.582913 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.582891    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:38:39.583791 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.583777    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:38:39.584505 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.584490    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.585280 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.585265    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:38:39.585946 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.585926    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wt2hf\""
Sep 17 15:38:39.586102 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.586090    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:38:39.586640 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.586625    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:39.586713 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.586686    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:38:39.587028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.587009    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:38:39.588236 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588219    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m8hjl\""
Sep 17 15:38:39.588236 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588228    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:38:39.588342 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588302    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:38:39.588342 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588325    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:38:39.588342 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588326    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:38:39.588483 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588369    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:38:39.588483 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588433    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33b10e70-61bd-4101-9a26-e0349b249102-host\") pod \"node-ca-4k758\" (UID: \"33b10e70-61bd-4101-9a26-e0349b249102\") " pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.588483 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588455    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmp97\" (UniqueName: \"kubernetes.io/projected/33b10e70-61bd-4101-9a26-e0349b249102-kube-api-access-xmp97\") pod \"node-ca-4k758\" (UID: \"33b10e70-61bd-4101-9a26-e0349b249102\") " pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.588585 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588478    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.588585 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588516    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d538ca1-32a3-4bd3-95f4-44177e3d30ab-tmp-dir\") pod \"node-resolver-rjjjt\" (UID: \"6d538ca1-32a3-4bd3-95f4-44177e3d30ab\") " pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.588585 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588568    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-run-netns\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.588675 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588591    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-node-log\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.588675 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588625    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-cni-netd\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.588675 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588647    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab9584e9-4c5a-4603-af6b-e26845d8bf04-ovn-node-metrics-cert\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.588767 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-sys-fs\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.588767 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588700    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkz85\" (UniqueName: \"kubernetes.io/projected/6d538ca1-32a3-4bd3-95f4-44177e3d30ab-kube-api-access-wkz85\") pod \"node-resolver-rjjjt\" (UID: \"6d538ca1-32a3-4bd3-95f4-44177e3d30ab\") " pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.588767 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588716    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5ff\" (UniqueName: \"kubernetes.io/projected/2cf8474f-71ab-45db-bfaa-e4953f520025-kube-api-access-mx5ff\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:39.588767 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-run-ovn-kubernetes\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.588767 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-cni-bin\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588779    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kv4d\" (UniqueName: \"kubernetes.io/projected/ab9584e9-4c5a-4603-af6b-e26845d8bf04-kube-api-access-2kv4d\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-cnibin\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588808    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-hostroot\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588845    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-run-multus-certs\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588875    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-etc-kubernetes\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588893    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-device-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588910    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-kubelet\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-etc-openvswitch\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588949    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-run-ovn\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588963    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab9584e9-4c5a-4603-af6b-e26845d8bf04-env-overrides\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.588977    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-system-cni-dir\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589000    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-run-k8s-cni-cncf-io\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589013    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp2xm\" (UniqueName: \"kubernetes.io/projected/26d5cfe5-3511-43d4-b487-1347b750a7f5-kube-api-access-gp2xm\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589025    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.589021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589027    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d538ca1-32a3-4bd3-95f4-44177e3d30ab-hosts-file\") pod \"node-resolver-rjjjt\" (UID: \"6d538ca1-32a3-4bd3-95f4-44177e3d30ab\") " pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589048    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-run-netns\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589082    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33b10e70-61bd-4101-9a26-e0349b249102-serviceca\") pod \"node-ca-4k758\" (UID: \"33b10e70-61bd-4101-9a26-e0349b249102\") " pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589107    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589129    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-registration-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589155    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589179    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-run-systemd\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589232    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26d5cfe5-3511-43d4-b487-1347b750a7f5-cni-binary-copy\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589248    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-var-lib-cni-bin\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589270    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-var-lib-kubelet\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589301    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-systemd-units\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-cni-dir\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-var-lib-cni-multus\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589383    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-conf-dir\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589410    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-daemon-config\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.589546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589432    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-socket-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.590028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589452    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-log-socket\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.590028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589480    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-run-openvswitch\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.590028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589502    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab9584e9-4c5a-4603-af6b-e26845d8bf04-ovnkube-config\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.590028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589521    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-socket-dir-parent\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.590028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589549    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdhjw\" (UniqueName: \"kubernetes.io/projected/6600b1ea-0e03-4858-9e42-77e0779cd3bd-kube-api-access-qdhjw\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.590028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589563    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-slash\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.590028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589579    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-var-lib-openvswitch\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.590028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589600    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab9584e9-4c5a-4603-af6b-e26845d8bf04-ovnkube-script-lib\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.590028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.589620    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz\") pod \"network-check-target-6nj4l\" (UID: \"01547a09-a86f-4c51-989e-e4d46ad48c5f\") " pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:39.591348 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.591330    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:38:39.591569 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.591552    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bdc4r\""
Sep 17 15:38:39.591889 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.591866    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:38:39.592196 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.592180    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:38:39.593951 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.593641    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.593951 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.593655    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lv52q\""
Sep 17 15:38:39.594910 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.594489    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:38:39.594910 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.594609    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:38:39.596369 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.595719    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:38:39.597621 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.597459    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:39.597982 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.597964    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:38:39.598066 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.597964    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:38:39.598066 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.598024    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v2ldn\""
Sep 17 15:38:39.604217 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.604196    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:38:39.604307 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.604198    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:38:39.604307 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.604251    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-69wmc\""
Sep 17 15:38:39.604484 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.604470    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:38:39.673903 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.673880    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m6zh2"
Sep 17 15:38:39.682410 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.682347    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m6zh2"
Sep 17 15:38:39.683928 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.683908    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:38:39.689789 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.689762    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-socket-dir-parent\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.690012 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.689793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-sysctl-d\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.690012 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.689819    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ee4a7c-2b79-4c3f-9405-c67719f47c54-iptables-alerter-script\") pod \"iptables-alerter-vqm4m\" (UID: \"c1ee4a7c-2b79-4c3f-9405-c67719f47c54\") " pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:39.690012 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.689883    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-socket-dir-parent\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.690012 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.689894    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-var-lib-openvswitch\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.690012 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.689935    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz\") pod \"network-check-target-6nj4l\" (UID: \"01547a09-a86f-4c51-989e-e4d46ad48c5f\") " pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:39.690012 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.689965    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33b10e70-61bd-4101-9a26-e0349b249102-host\") pod \"node-ca-4k758\" (UID: \"33b10e70-61bd-4101-9a26-e0349b249102\") " pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.690012 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.689974    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-var-lib-openvswitch\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.690012 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.689993    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmp97\" (UniqueName: \"kubernetes.io/projected/33b10e70-61bd-4101-9a26-e0349b249102-kube-api-access-xmp97\") pod \"node-ca-4k758\" (UID: \"33b10e70-61bd-4101-9a26-e0349b249102\") " pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690019    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690045    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d538ca1-32a3-4bd3-95f4-44177e3d30ab-tmp-dir\") pod \"node-resolver-rjjjt\" (UID: \"6d538ca1-32a3-4bd3-95f4-44177e3d30ab\") " pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690074    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c611bed6-3ca1-412d-9047-b574034b42d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690099    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-var-lib-kubelet\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690124    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-cni-netd\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690149    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab9584e9-4c5a-4603-af6b-e26845d8bf04-ovn-node-metrics-cert\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690173    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-sysconfig\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690205    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ee4a7c-2b79-4c3f-9405-c67719f47c54-host-slash\") pod \"iptables-alerter-vqm4m\" (UID: \"c1ee4a7c-2b79-4c3f-9405-c67719f47c54\") " pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690250    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-cni-bin\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690266    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-cni-netd\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690274    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-cnibin\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-run-multus-certs\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690333    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-etc-kubernetes\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690360    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c611bed6-3ca1-412d-9047-b574034b42d8-cnibin\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690398    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6acb82a-8eb6-4508-9345-88b203e27b0e-kubelet-config\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:39.690688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690425    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690452    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e6f463-8129-42ff-b9a1-3622dfaa7163-tmp\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690477    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-kubelet\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690503    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-run-ovn\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d538ca1-32a3-4bd3-95f4-44177e3d30ab-tmp-dir\") pod \"node-resolver-rjjjt\" (UID: \"6d538ca1-32a3-4bd3-95f4-44177e3d30ab\") " pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab9584e9-4c5a-4603-af6b-e26845d8bf04-env-overrides\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690595    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvgtv\" (UniqueName: \"kubernetes.io/projected/65e6f463-8129-42ff-b9a1-3622dfaa7163-kube-api-access-gvgtv\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690637    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-run-netns\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690657    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33b10e70-61bd-4101-9a26-e0349b249102-host\") pod \"node-ca-4k758\" (UID: \"33b10e70-61bd-4101-9a26-e0349b249102\") " pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690662    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-registration-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c611bed6-3ca1-412d-9047-b574034b42d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690730    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-run-multus-certs\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-sysctl-conf\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690771    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-run-netns\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690786    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-registration-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690818    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-cnibin\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.691504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690819    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26d5cfe5-3511-43d4-b487-1347b750a7f5-cni-binary-copy\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690736    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-cni-bin\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-kubelet\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-var-lib-kubelet\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690889    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-etc-kubernetes\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690897    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690911    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-run-ovn\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690908    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2t95\" (UniqueName: \"kubernetes.io/projected/c611bed6-3ca1-412d-9047-b574034b42d8-kube-api-access-l2t95\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690938    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-var-lib-kubelet\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690962    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-cni-dir\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691053    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab9584e9-4c5a-4603-af6b-e26845d8bf04-env-overrides\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-conf-dir\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691133    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-cni-dir\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691221    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.690986    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-conf-dir\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691325    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-daemon-config\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691354    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-socket-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691398    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdhjw\" (UniqueName: \"kubernetes.io/projected/6600b1ea-0e03-4858-9e42-77e0779cd3bd-kube-api-access-qdhjw\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.692390 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c611bed6-3ca1-412d-9047-b574034b42d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691453    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6acb82a-8eb6-4508-9345-88b203e27b0e-dbus\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691468    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26d5cfe5-3511-43d4-b487-1347b750a7f5-cni-binary-copy\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691476    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-log-socket\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691498    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab9584e9-4c5a-4603-af6b-e26845d8bf04-ovnkube-config\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691520    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd89a8d5-387c-4f46-80d2-1c7ad2bc8286-konnectivity-ca\") pod \"konnectivity-agent-p65h2\" (UID: \"fd89a8d5-387c-4f46-80d2-1c7ad2bc8286\") " pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-slash\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691564    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab9584e9-4c5a-4603-af6b-e26845d8bf04-ovnkube-script-lib\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691587    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkz85\" (UniqueName: \"kubernetes.io/projected/6d538ca1-32a3-4bd3-95f4-44177e3d30ab-kube-api-access-wkz85\") pod \"node-resolver-rjjjt\" (UID: \"6d538ca1-32a3-4bd3-95f4-44177e3d30ab\") " pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691610    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-modprobe-d\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691632    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-systemd\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691637    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-log-socket\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691656    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-run\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691682    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-run-netns\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691733    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-node-log\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691756    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-sys-fs\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691780    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5ff\" (UniqueName: \"kubernetes.io/projected/2cf8474f-71ab-45db-bfaa-e4953f520025-kube-api-access-mx5ff\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:39.693221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-socket-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691841    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-kubernetes\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691865    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-host\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691888    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-run-ovn-kubernetes\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691914    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv4d\" (UniqueName: \"kubernetes.io/projected/ab9584e9-4c5a-4603-af6b-e26845d8bf04-kube-api-access-2kv4d\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691972    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-hostroot\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.691999    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-device-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692027    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d538ca1-32a3-4bd3-95f4-44177e3d30ab-hosts-file\") pod \"node-resolver-rjjjt\" (UID: \"6d538ca1-32a3-4bd3-95f4-44177e3d30ab\") " pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c611bed6-3ca1-412d-9047-b574034b42d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692080    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-etc-openvswitch\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692105    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-system-cni-dir\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692107    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-slash\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-run-k8s-cni-cncf-io\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692174    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp2xm\" (UniqueName: \"kubernetes.io/projected/26d5cfe5-3511-43d4-b487-1347b750a7f5-kube-api-access-gp2xm\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692209    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33b10e70-61bd-4101-9a26-e0349b249102-serviceca\") pod \"node-ca-4k758\" (UID: \"33b10e70-61bd-4101-9a26-e0349b249102\") " pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692233    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692260    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:39.693777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692280    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26d5cfe5-3511-43d4-b487-1347b750a7f5-multus-daemon-config\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-run-systemd\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692327    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-var-lib-cni-bin\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-lib-modules\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692380    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-tuned\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692406    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-systemd-units\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692436    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-var-lib-cni-multus\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692461    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c611bed6-3ca1-412d-9047-b574034b42d8-system-cni-dir\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692493    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-sys\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bz4n\" (UniqueName: \"kubernetes.io/projected/c1ee4a7c-2b79-4c3f-9405-c67719f47c54-kube-api-access-8bz4n\") pod \"iptables-alerter-vqm4m\" (UID: \"c1ee4a7c-2b79-4c3f-9405-c67719f47c54\") " pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692539    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd89a8d5-387c-4f46-80d2-1c7ad2bc8286-agent-certs\") pod \"konnectivity-agent-p65h2\" (UID: \"fd89a8d5-387c-4f46-80d2-1c7ad2bc8286\") " pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692541    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-run-ovn-kubernetes\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692580    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-run-openvswitch\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692626    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d538ca1-32a3-4bd3-95f4-44177e3d30ab-hosts-file\") pod \"node-resolver-rjjjt\" (UID: \"6d538ca1-32a3-4bd3-95f4-44177e3d30ab\") " pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692632    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab9584e9-4c5a-4603-af6b-e26845d8bf04-ovnkube-script-lib\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692699    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-hostroot\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692703    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-host-run-netns\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694299 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692698    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-sys-fs\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692740    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-node-log\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692746    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-etc-openvswitch\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692769    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-run-openvswitch\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692796    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-run-systemd\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-device-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692174    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab9584e9-4c5a-4603-af6b-e26845d8bf04-ovnkube-config\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692847    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-run-k8s-cni-cncf-io\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692845    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-var-lib-cni-multus\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692898    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab9584e9-4c5a-4603-af6b-e26845d8bf04-systemd-units\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.692934    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.692966    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-system-cni-dir\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.693017    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26d5cfe5-3511-43d4-b487-1347b750a7f5-host-var-lib-cni-bin\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.693030    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs podName:2cf8474f-71ab-45db-bfaa-e4953f520025 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:40.192988967 +0000 UTC m=+2.083849489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs") pod "network-metrics-daemon-b2879" (UID: "2cf8474f-71ab-45db-bfaa-e4953f520025") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.693019    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6600b1ea-0e03-4858-9e42-77e0779cd3bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.694853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.693346    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33b10e70-61bd-4101-9a26-e0349b249102-serviceca\") pod \"node-ca-4k758\" (UID: \"33b10e70-61bd-4101-9a26-e0349b249102\") " pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.695284 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.694966    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab9584e9-4c5a-4603-af6b-e26845d8bf04-ovn-node-metrics-cert\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.696539 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.696516    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:39.696539 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.696539    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:39.696677 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.696549    2555 projected.go:196] Error preparing data for projected volume kube-api-access-nv6sz for pod openshift-network-diagnostics/network-check-target-6nj4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:39.696677 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.696607    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz podName:01547a09-a86f-4c51-989e-e4d46ad48c5f nodeName:}" failed. No retries permitted until 2026-09-17 15:38:40.196589615 +0000 UTC m=+2.087450162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nv6sz" (UniqueName: "kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz") pod "network-check-target-6nj4l" (UID: "01547a09-a86f-4c51-989e-e4d46ad48c5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:39.698846 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.698804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmp97\" (UniqueName: \"kubernetes.io/projected/33b10e70-61bd-4101-9a26-e0349b249102-kube-api-access-xmp97\") pod \"node-ca-4k758\" (UID: \"33b10e70-61bd-4101-9a26-e0349b249102\") " pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.704468 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.704437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkz85\" (UniqueName: \"kubernetes.io/projected/6d538ca1-32a3-4bd3-95f4-44177e3d30ab-kube-api-access-wkz85\") pod \"node-resolver-rjjjt\" (UID: \"6d538ca1-32a3-4bd3-95f4-44177e3d30ab\") " pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.706518 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.706481    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdhjw\" (UniqueName: \"kubernetes.io/projected/6600b1ea-0e03-4858-9e42-77e0779cd3bd-kube-api-access-qdhjw\") pod \"aws-ebs-csi-driver-node-9x9ss\" (UID: \"6600b1ea-0e03-4858-9e42-77e0779cd3bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.706621 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.706593    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5ff\" (UniqueName: \"kubernetes.io/projected/2cf8474f-71ab-45db-bfaa-e4953f520025-kube-api-access-mx5ff\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:39.706963 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.706942    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv4d\" (UniqueName: \"kubernetes.io/projected/ab9584e9-4c5a-4603-af6b-e26845d8bf04-kube-api-access-2kv4d\") pod \"ovnkube-node-h7dwx\" (UID: \"ab9584e9-4c5a-4603-af6b-e26845d8bf04\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.707446 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.707426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp2xm\" (UniqueName: \"kubernetes.io/projected/26d5cfe5-3511-43d4-b487-1347b750a7f5-kube-api-access-gp2xm\") pod \"multus-gxshr\" (UID: \"26d5cfe5-3511-43d4-b487-1347b750a7f5\") " pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.793148 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793103    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-lib-modules\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793158    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-tuned\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793188    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c611bed6-3ca1-412d-9047-b574034b42d8-system-cni-dir\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.793350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793212    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-sys\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bz4n\" (UniqueName: \"kubernetes.io/projected/c1ee4a7c-2b79-4c3f-9405-c67719f47c54-kube-api-access-8bz4n\") pod \"iptables-alerter-vqm4m\" (UID: \"c1ee4a7c-2b79-4c3f-9405-c67719f47c54\") " pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:39.793350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793256    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd89a8d5-387c-4f46-80d2-1c7ad2bc8286-agent-certs\") pod \"konnectivity-agent-p65h2\" (UID: \"fd89a8d5-387c-4f46-80d2-1c7ad2bc8286\") " pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:38:39.793350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793279    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-sysctl-d\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793301    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ee4a7c-2b79-4c3f-9405-c67719f47c54-iptables-alerter-script\") pod \"iptables-alerter-vqm4m\" (UID: \"c1ee4a7c-2b79-4c3f-9405-c67719f47c54\") " pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:39.793350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793327    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-sys\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793342    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c611bed6-3ca1-412d-9047-b574034b42d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793366    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-var-lib-kubelet\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793363    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-lib-modules\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-sysconfig\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793439    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-var-lib-kubelet\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ee4a7c-2b79-4c3f-9405-c67719f47c54-host-slash\") pod \"iptables-alerter-vqm4m\" (UID: \"c1ee4a7c-2b79-4c3f-9405-c67719f47c54\") " pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793485    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c611bed6-3ca1-412d-9047-b574034b42d8-cnibin\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793510    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6acb82a-8eb6-4508-9345-88b203e27b0e-kubelet-config\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793512    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-sysctl-d\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793518    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ee4a7c-2b79-4c3f-9405-c67719f47c54-host-slash\") pod \"iptables-alerter-vqm4m\" (UID: \"c1ee4a7c-2b79-4c3f-9405-c67719f47c54\") " pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793534    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793559    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-sysconfig\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793565    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e6f463-8129-42ff-b9a1-3622dfaa7163-tmp\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793601    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c611bed6-3ca1-412d-9047-b574034b42d8-cnibin\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6acb82a-8eb6-4508-9345-88b203e27b0e-kubelet-config\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793627    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvgtv\" (UniqueName: \"kubernetes.io/projected/65e6f463-8129-42ff-b9a1-3622dfaa7163-kube-api-access-gvgtv\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793658    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c611bed6-3ca1-412d-9047-b574034b42d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.793683    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:39.793884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-sysctl-conf\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793740    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2t95\" (UniqueName: \"kubernetes.io/projected/c611bed6-3ca1-412d-9047-b574034b42d8-kube-api-access-l2t95\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:39.793757    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret podName:a6acb82a-8eb6-4508-9345-88b203e27b0e nodeName:}" failed. No retries permitted until 2026-09-17 15:38:40.293736255 +0000 UTC m=+2.184596784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret") pod "global-pull-secret-syncer-nlf6d" (UID: "a6acb82a-8eb6-4508-9345-88b203e27b0e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793796    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c611bed6-3ca1-412d-9047-b574034b42d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6acb82a-8eb6-4508-9345-88b203e27b0e-dbus\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793870    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd89a8d5-387c-4f46-80d2-1c7ad2bc8286-konnectivity-ca\") pod \"konnectivity-agent-p65h2\" (UID: \"fd89a8d5-387c-4f46-80d2-1c7ad2bc8286\") " pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793919    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-modprobe-d\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793941    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-systemd\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.793963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-run\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794003    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ee4a7c-2b79-4c3f-9405-c67719f47c54-iptables-alerter-script\") pod \"iptables-alerter-vqm4m\" (UID: \"c1ee4a7c-2b79-4c3f-9405-c67719f47c54\") " pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794073    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6acb82a-8eb6-4508-9345-88b203e27b0e-dbus\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c611bed6-3ca1-412d-9047-b574034b42d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794009    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-kubernetes\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794204    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-host\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794228    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c611bed6-3ca1-412d-9047-b574034b42d8-system-cni-dir\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794239    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c611bed6-3ca1-412d-9047-b574034b42d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.794752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-host\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.795598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794346    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-systemd\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.795598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794442    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-sysctl-conf\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.795598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794453    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-modprobe-d\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.795598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794524    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-run\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.795598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794571    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-kubernetes\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.795598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794685    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd89a8d5-387c-4f46-80d2-1c7ad2bc8286-konnectivity-ca\") pod \"konnectivity-agent-p65h2\" (UID: \"fd89a8d5-387c-4f46-80d2-1c7ad2bc8286\") " pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:38:39.795598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794848    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c611bed6-3ca1-412d-9047-b574034b42d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.795598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.794909    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c611bed6-3ca1-412d-9047-b574034b42d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.795598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.795071    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c611bed6-3ca1-412d-9047-b574034b42d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.796060 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.796042    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65e6f463-8129-42ff-b9a1-3622dfaa7163-etc-tuned\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.796273 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.796257    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e6f463-8129-42ff-b9a1-3622dfaa7163-tmp\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.796429 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.796414    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd89a8d5-387c-4f46-80d2-1c7ad2bc8286-agent-certs\") pod \"konnectivity-agent-p65h2\" (UID: \"fd89a8d5-387c-4f46-80d2-1c7ad2bc8286\") " pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:38:39.802357 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.802325    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2t95\" (UniqueName: \"kubernetes.io/projected/c611bed6-3ca1-412d-9047-b574034b42d8-kube-api-access-l2t95\") pod \"multus-additional-cni-plugins-pjwg8\" (UID: \"c611bed6-3ca1-412d-9047-b574034b42d8\") " pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:39.803050 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.803029    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bz4n\" (UniqueName: \"kubernetes.io/projected/c1ee4a7c-2b79-4c3f-9405-c67719f47c54-kube-api-access-8bz4n\") pod \"iptables-alerter-vqm4m\" (UID: \"c1ee4a7c-2b79-4c3f-9405-c67719f47c54\") " pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:39.805428 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.805402    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvgtv\" (UniqueName: \"kubernetes.io/projected/65e6f463-8129-42ff-b9a1-3622dfaa7163-kube-api-access-gvgtv\") pod \"tuned-phv6p\" (UID: \"65e6f463-8129-42ff-b9a1-3622dfaa7163\") " pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.853251 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:39.853206    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff4a4436cf19c431e9ef9db87e16884.slice/crio-a87db5895ac854c94fbbbaf6ed9f59dc39a8b3a0c4411a2c19150d6d3bb6c7e6 WatchSource:0}: Error finding container a87db5895ac854c94fbbbaf6ed9f59dc39a8b3a0c4411a2c19150d6d3bb6c7e6: Status 404 returned error can't find the container with id a87db5895ac854c94fbbbaf6ed9f59dc39a8b3a0c4411a2c19150d6d3bb6c7e6
Sep 17 15:38:39.857398 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.857379    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:38:39.909136 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.909093    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4k758"
Sep 17 15:38:39.915727 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.915707    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss"
Sep 17 15:38:39.920638 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:39.920604    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b10e70_61bd_4101_9a26_e0349b249102.slice/crio-945b56166b0330504f5e429f1fe3784b425d84c49cad20208190b38e07860ce6 WatchSource:0}: Error finding container 945b56166b0330504f5e429f1fe3784b425d84c49cad20208190b38e07860ce6: Status 404 returned error can't find the container with id 945b56166b0330504f5e429f1fe3784b425d84c49cad20208190b38e07860ce6
Sep 17 15:38:39.925785 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:39.925759    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6600b1ea_0e03_4858_9e42_77e0779cd3bd.slice/crio-5d69f59f2164ec11fed6c658813e07ae9db88509628e8100882a4d4e0af04c66 WatchSource:0}: Error finding container 5d69f59f2164ec11fed6c658813e07ae9db88509628e8100882a4d4e0af04c66: Status 404 returned error can't find the container with id 5d69f59f2164ec11fed6c658813e07ae9db88509628e8100882a4d4e0af04c66
Sep 17 15:38:39.948925 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.948838    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rjjjt"
Sep 17 15:38:39.958405 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:39.958361    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d538ca1_32a3_4bd3_95f4_44177e3d30ab.slice/crio-693ed38aaeaac5c31a95c4767b7b3dd0840e9c7e043a3786a18790d1cf9efff2 WatchSource:0}: Error finding container 693ed38aaeaac5c31a95c4767b7b3dd0840e9c7e043a3786a18790d1cf9efff2: Status 404 returned error can't find the container with id 693ed38aaeaac5c31a95c4767b7b3dd0840e9c7e043a3786a18790d1cf9efff2
Sep 17 15:38:39.966762 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.966730    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gxshr"
Sep 17 15:38:39.971903 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.971883    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:38:39.976073 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:39.976028    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d5cfe5_3511_43d4_b487_1347b750a7f5.slice/crio-113d0b19c7ecd6a72115a5fb725dbfed8444617a5a88e4453461ee01cc492f23 WatchSource:0}: Error finding container 113d0b19c7ecd6a72115a5fb725dbfed8444617a5a88e4453461ee01cc492f23: Status 404 returned error can't find the container with id 113d0b19c7ecd6a72115a5fb725dbfed8444617a5a88e4453461ee01cc492f23
Sep 17 15:38:39.980860 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:39.980819    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9584e9_4c5a_4603_af6b_e26845d8bf04.slice/crio-eb292b6686ef693f977767cd2ad65d0ce0a3cd6b3996aa27ded8ecba1b87feaa WatchSource:0}: Error finding container eb292b6686ef693f977767cd2ad65d0ce0a3cd6b3996aa27ded8ecba1b87feaa: Status 404 returned error can't find the container with id eb292b6686ef693f977767cd2ad65d0ce0a3cd6b3996aa27ded8ecba1b87feaa
Sep 17 15:38:39.987230 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:39.987209    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-phv6p"
Sep 17 15:38:39.996561 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:39.996537    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e6f463_8129_42ff_b9a1_3622dfaa7163.slice/crio-ae2ce4f1e8be6b0779cadd5bcb939619c19ba19670699c0e90acd2ed677192ad WatchSource:0}: Error finding container ae2ce4f1e8be6b0779cadd5bcb939619c19ba19670699c0e90acd2ed677192ad: Status 404 returned error can't find the container with id ae2ce4f1e8be6b0779cadd5bcb939619c19ba19670699c0e90acd2ed677192ad
Sep 17 15:38:40.014792 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.014757    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:38:40.022003 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.021978    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pjwg8"
Sep 17 15:38:40.023761 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:40.023734    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd89a8d5_387c_4f46_80d2_1c7ad2bc8286.slice/crio-dc9317996a4413aaaaa2dfec7acf53ea11566678e50a0c9e940326c834ec4b7b WatchSource:0}: Error finding container dc9317996a4413aaaaa2dfec7acf53ea11566678e50a0c9e940326c834ec4b7b: Status 404 returned error can't find the container with id dc9317996a4413aaaaa2dfec7acf53ea11566678e50a0c9e940326c834ec4b7b
Sep 17 15:38:40.027245 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.027171    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vqm4m"
Sep 17 15:38:40.031088 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:40.031053    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc611bed6_3ca1_412d_9047_b574034b42d8.slice/crio-0f83272a9f2c15fec46ef1a826acf21e7baeaf670513ac7b0d7a5150ddaf1a9b WatchSource:0}: Error finding container 0f83272a9f2c15fec46ef1a826acf21e7baeaf670513ac7b0d7a5150ddaf1a9b: Status 404 returned error can't find the container with id 0f83272a9f2c15fec46ef1a826acf21e7baeaf670513ac7b0d7a5150ddaf1a9b
Sep 17 15:38:40.037366 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:38:40.037336    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ee4a7c_2b79_4c3f_9405_c67719f47c54.slice/crio-feffa650cd90dfdc3490c76e324b001c16cbb34981fe1b23b0207db39bdafd47 WatchSource:0}: Error finding container feffa650cd90dfdc3490c76e324b001c16cbb34981fe1b23b0207db39bdafd47: Status 404 returned error can't find the container with id feffa650cd90dfdc3490c76e324b001c16cbb34981fe1b23b0207db39bdafd47
Sep 17 15:38:40.039310 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.039284    2555 kuberuntime_manager.go:1459] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-vqm4m_openshift-network-operator(c1ee4a7c-2b79-4c3f-9405-c67719f47c54): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 17 15:38:40.039367 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.039331    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-vqm4m" podUID="c1ee4a7c-2b79-4c3f-9405-c67719f47c54"
Sep 17 15:38:40.196381 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.196343    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:40.196533 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.196492    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:40.196567 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.196561    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs podName:2cf8474f-71ab-45db-bfaa-e4953f520025 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:41.196546243 +0000 UTC m=+3.087406764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs") pod "network-metrics-daemon-b2879" (UID: "2cf8474f-71ab-45db-bfaa-e4953f520025") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:40.297957 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.297587    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz\") pod \"network-check-target-6nj4l\" (UID: \"01547a09-a86f-4c51-989e-e4d46ad48c5f\") " pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:40.297957 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.297950    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:40.298252 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.297788    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:40.298252 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.298009    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:40.298252 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.298028    2555 projected.go:196] Error preparing data for projected volume kube-api-access-nv6sz for pod openshift-network-diagnostics/network-check-target-6nj4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:40.298252 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.298103    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz podName:01547a09-a86f-4c51-989e-e4d46ad48c5f nodeName:}" failed. No retries permitted until 2026-09-17 15:38:41.298081566 +0000 UTC m=+3.188942099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv6sz" (UniqueName: "kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz") pod "network-check-target-6nj4l" (UID: "01547a09-a86f-4c51-989e-e4d46ad48c5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:40.298252 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.298115    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:40.298252 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.298168    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret podName:a6acb82a-8eb6-4508-9345-88b203e27b0e nodeName:}" failed. No retries permitted until 2026-09-17 15:38:41.298150967 +0000 UTC m=+3.189011493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret") pod "global-pull-secret-syncer-nlf6d" (UID: "a6acb82a-8eb6-4508-9345-88b203e27b0e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:40.302030 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.302003    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:40.686510 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.686391    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:33:39 +0000 UTC" deadline="2028-04-30 20:35:49.546239754 +0000 UTC"
Sep 17 15:38:40.686510 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.686457    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14188h57m8.85978617s"
Sep 17 15:38:40.688787 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.688748    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:40.689121 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.688938    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:40.728781 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.728714    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vqm4m" event={"ID":"c1ee4a7c-2b79-4c3f-9405-c67719f47c54","Type":"ContainerStarted","Data":"feffa650cd90dfdc3490c76e324b001c16cbb34981fe1b23b0207db39bdafd47"}
Sep 17 15:38:40.734846 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:40.734795    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0444a9580f430c7c6b52f78667a80a82198c825805f1f2004fdfdf9aed2b95f5\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-vqm4m" podUID="c1ee4a7c-2b79-4c3f-9405-c67719f47c54"
Sep 17 15:38:40.746875 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.746834    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gxshr" event={"ID":"26d5cfe5-3511-43d4-b487-1347b750a7f5","Type":"ContainerStarted","Data":"113d0b19c7ecd6a72115a5fb725dbfed8444617a5a88e4453461ee01cc492f23"}
Sep 17 15:38:40.755308 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.755277    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:40.757545 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.757512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss" event={"ID":"6600b1ea-0e03-4858-9e42-77e0779cd3bd","Type":"ContainerStarted","Data":"5d69f59f2164ec11fed6c658813e07ae9db88509628e8100882a4d4e0af04c66"}
Sep 17 15:38:40.790113 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.790022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4k758" event={"ID":"33b10e70-61bd-4101-9a26-e0349b249102","Type":"ContainerStarted","Data":"945b56166b0330504f5e429f1fe3784b425d84c49cad20208190b38e07860ce6"}
Sep 17 15:38:40.809342 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.809300    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal" event={"ID":"eff4a4436cf19c431e9ef9db87e16884","Type":"ContainerStarted","Data":"a87db5895ac854c94fbbbaf6ed9f59dc39a8b3a0c4411a2c19150d6d3bb6c7e6"}
Sep 17 15:38:40.831774 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.831733    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjwg8" event={"ID":"c611bed6-3ca1-412d-9047-b574034b42d8","Type":"ContainerStarted","Data":"0f83272a9f2c15fec46ef1a826acf21e7baeaf670513ac7b0d7a5150ddaf1a9b"}
Sep 17 15:38:40.860586 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.860540    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p65h2" event={"ID":"fd89a8d5-387c-4f46-80d2-1c7ad2bc8286","Type":"ContainerStarted","Data":"dc9317996a4413aaaaa2dfec7acf53ea11566678e50a0c9e940326c834ec4b7b"}
Sep 17 15:38:40.881832 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.881792    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-phv6p" event={"ID":"65e6f463-8129-42ff-b9a1-3622dfaa7163","Type":"ContainerStarted","Data":"ae2ce4f1e8be6b0779cadd5bcb939619c19ba19670699c0e90acd2ed677192ad"}
Sep 17 15:38:40.908565 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.908525    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" event={"ID":"ab9584e9-4c5a-4603-af6b-e26845d8bf04","Type":"ContainerStarted","Data":"eb292b6686ef693f977767cd2ad65d0ce0a3cd6b3996aa27ded8ecba1b87feaa"}
Sep 17 15:38:40.929761 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.929714    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rjjjt" event={"ID":"6d538ca1-32a3-4bd3-95f4-44177e3d30ab","Type":"ContainerStarted","Data":"693ed38aaeaac5c31a95c4767b7b3dd0840e9c7e043a3786a18790d1cf9efff2"}
Sep 17 15:38:40.939270 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.939174    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:40.942121 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:40.941871    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal" event={"ID":"8ce232a731178d99d36cb8d051eb3270","Type":"ContainerStarted","Data":"7939e3abddd1cd5bd60527ba410187de6df4d9a58c931691b4dd49279ae2aa38"}
Sep 17 15:38:41.206881 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:41.206120    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:41.206881 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:41.206326    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:41.206881 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:41.206403    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs podName:2cf8474f-71ab-45db-bfaa-e4953f520025 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:43.2063821 +0000 UTC m=+5.097242631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs") pod "network-metrics-daemon-b2879" (UID: "2cf8474f-71ab-45db-bfaa-e4953f520025") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:41.306979 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:41.306561    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz\") pod \"network-check-target-6nj4l\" (UID: \"01547a09-a86f-4c51-989e-e4d46ad48c5f\") " pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:41.306979 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:41.306628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:41.306979 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:41.306762    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:41.306979 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:41.306833    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret podName:a6acb82a-8eb6-4508-9345-88b203e27b0e nodeName:}" failed. No retries permitted until 2026-09-17 15:38:43.306810396 +0000 UTC m=+5.197670925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret") pod "global-pull-secret-syncer-nlf6d" (UID: "a6acb82a-8eb6-4508-9345-88b203e27b0e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:41.307370 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:41.307246    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:41.307370 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:41.307268    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:41.307370 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:41.307284    2555 projected.go:196] Error preparing data for projected volume kube-api-access-nv6sz for pod openshift-network-diagnostics/network-check-target-6nj4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:41.307370 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:41.307332    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz podName:01547a09-a86f-4c51-989e-e4d46ad48c5f nodeName:}" failed. No retries permitted until 2026-09-17 15:38:43.307316386 +0000 UTC m=+5.198176913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv6sz" (UniqueName: "kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz") pod "network-check-target-6nj4l" (UID: "01547a09-a86f-4c51-989e-e4d46ad48c5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:41.687581 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:41.687475    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:33:39 +0000 UTC" deadline="2028-06-03 23:06:49.834146282 +0000 UTC"
Sep 17 15:38:41.687581 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:41.687524    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15007h28m8.14662793s"
Sep 17 15:38:41.688828 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:41.688802    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:41.688986 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:41.688959    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:41.689343 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:41.689322    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:41.689457 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:41.689434    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:38:42.691629 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:42.691596    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:42.692122 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:42.691736    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:43.225772 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:43.225678    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:43.225971 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:43.225925    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:43.226031 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:43.225993    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs podName:2cf8474f-71ab-45db-bfaa-e4953f520025 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:47.225971827 +0000 UTC m=+9.116832363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs") pod "network-metrics-daemon-b2879" (UID: "2cf8474f-71ab-45db-bfaa-e4953f520025") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:43.326223 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:43.326158    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz\") pod \"network-check-target-6nj4l\" (UID: \"01547a09-a86f-4c51-989e-e4d46ad48c5f\") " pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:43.326223 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:43.326215    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:43.326482 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:43.326281    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:43.326482 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:43.326298    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:43.326482 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:43.326309    2555 projected.go:196] Error preparing data for projected volume kube-api-access-nv6sz for pod openshift-network-diagnostics/network-check-target-6nj4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:43.326482 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:43.326349    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:43.326482 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:43.326353    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz podName:01547a09-a86f-4c51-989e-e4d46ad48c5f nodeName:}" failed. No retries permitted until 2026-09-17 15:38:47.326338364 +0000 UTC m=+9.217198885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv6sz" (UniqueName: "kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz") pod "network-check-target-6nj4l" (UID: "01547a09-a86f-4c51-989e-e4d46ad48c5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:43.326482 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:43.326407    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret podName:a6acb82a-8eb6-4508-9345-88b203e27b0e nodeName:}" failed. No retries permitted until 2026-09-17 15:38:47.326394912 +0000 UTC m=+9.217255434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret") pod "global-pull-secret-syncer-nlf6d" (UID: "a6acb82a-8eb6-4508-9345-88b203e27b0e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:43.689285 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:43.689197    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:43.689285 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:43.689219    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:43.689529 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:43.689348    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:43.689529 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:43.689449    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:38:44.689852 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:44.689410    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:44.689852 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:44.689582    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:45.688997 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:45.688963    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:45.689175 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:45.689097    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:38:45.689175 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:45.689164    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:45.689314 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:45.689245    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:46.688682 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:46.688597    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:46.689230 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:46.688753    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:47.260645 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:47.260602    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:47.260801 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:47.260774    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:47.260918 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:47.260857    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs podName:2cf8474f-71ab-45db-bfaa-e4953f520025 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:55.260836507 +0000 UTC m=+17.151697043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs") pod "network-metrics-daemon-b2879" (UID: "2cf8474f-71ab-45db-bfaa-e4953f520025") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:47.362570 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:47.361755    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:47.362570 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:47.361869    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz\") pod \"network-check-target-6nj4l\" (UID: \"01547a09-a86f-4c51-989e-e4d46ad48c5f\") " pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:47.362570 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:47.362013    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:47.362570 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:47.362033    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:47.362570 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:47.362047    2555 projected.go:196] Error preparing data for projected volume kube-api-access-nv6sz for pod openshift-network-diagnostics/network-check-target-6nj4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:47.362570 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:47.362110    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz podName:01547a09-a86f-4c51-989e-e4d46ad48c5f nodeName:}" failed. No retries permitted until 2026-09-17 15:38:55.362091007 +0000 UTC m=+17.252951536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv6sz" (UniqueName: "kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz") pod "network-check-target-6nj4l" (UID: "01547a09-a86f-4c51-989e-e4d46ad48c5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:47.362570 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:47.362480    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:47.362570 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:47.362534    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret podName:a6acb82a-8eb6-4508-9345-88b203e27b0e nodeName:}" failed. No retries permitted until 2026-09-17 15:38:55.362517005 +0000 UTC m=+17.253377532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret") pod "global-pull-secret-syncer-nlf6d" (UID: "a6acb82a-8eb6-4508-9345-88b203e27b0e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:47.688943 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:47.688910    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:47.689557 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:47.688962    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:47.689557 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:47.689054    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:47.689557 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:47.689213    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:38:48.689798 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:48.689546    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:48.689798 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:48.689697    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:49.689052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:49.689016    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:49.689211 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:49.689014    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:49.689211 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:49.689132    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:38:49.689297 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:49.689208    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:50.689321 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:50.689280    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:50.689813 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:50.689409    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:51.689068 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:51.689028    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:51.689256 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:51.689166    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:51.689256 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:51.689206    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:51.689395 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:51.689325    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:38:52.689110 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:52.689069    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:52.689303 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:52.689231    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:53.688815 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:53.688775    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:53.688815 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:53.688811    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:53.689292 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:53.688934    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:38:53.689292 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:53.689040    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:54.689056 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:54.689016    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:54.689521 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:54.689169    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:55.324139 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:55.324101    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:55.324354 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:55.324260    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:55.324354 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:55.324334    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs podName:2cf8474f-71ab-45db-bfaa-e4953f520025 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:11.324315028 +0000 UTC m=+33.215175554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs") pod "network-metrics-daemon-b2879" (UID: "2cf8474f-71ab-45db-bfaa-e4953f520025") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:55.424720 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:55.424679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz\") pod \"network-check-target-6nj4l\" (UID: \"01547a09-a86f-4c51-989e-e4d46ad48c5f\") " pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:55.424925 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:55.424731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:55.424925 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:55.424858    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:55.424925 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:55.424877    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:55.424925 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:55.424900    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:55.424925 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:55.424911    2555 projected.go:196] Error preparing data for projected volume kube-api-access-nv6sz for pod openshift-network-diagnostics/network-check-target-6nj4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:55.424925 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:55.424921    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret podName:a6acb82a-8eb6-4508-9345-88b203e27b0e nodeName:}" failed. No retries permitted until 2026-09-17 15:39:11.424902745 +0000 UTC m=+33.315763266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret") pod "global-pull-secret-syncer-nlf6d" (UID: "a6acb82a-8eb6-4508-9345-88b203e27b0e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:55.425208 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:55.424977    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz podName:01547a09-a86f-4c51-989e-e4d46ad48c5f nodeName:}" failed. No retries permitted until 2026-09-17 15:39:11.424965151 +0000 UTC m=+33.315825674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv6sz" (UniqueName: "kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz") pod "network-check-target-6nj4l" (UID: "01547a09-a86f-4c51-989e-e4d46ad48c5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:55.689398 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:55.689303    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:55.689398 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:55.689362    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:55.689892 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:55.689452    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:55.689892 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:55.689557    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:38:56.688493 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:56.688455    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:56.688703 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:56.688604    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:57.689229 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:57.689191    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:57.689669 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:57.689193    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:57.689669 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:57.689339    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:38:57.689669 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:57.689489    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:58.688848 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:58.688810    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:38:58.688949 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:58.688910    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:38:59.004818 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:59.004616    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-phv6p" event={"ID":"65e6f463-8129-42ff-b9a1-3622dfaa7163","Type":"ContainerStarted","Data":"bfd95ca8523b6b58f863ca143a8c08c92a9ab070418332eaee982cb9a507dc87"}
Sep 17 15:38:59.007273 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:59.006921    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal" event={"ID":"8ce232a731178d99d36cb8d051eb3270","Type":"ContainerStarted","Data":"ca92b1561bdd4da6ce93ee9d21dd29229260ad25fd2d5e868a2f1dcc2eb62fbb"}
Sep 17 15:38:59.008949 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:59.008929    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gxshr" event={"ID":"26d5cfe5-3511-43d4-b487-1347b750a7f5","Type":"ContainerStarted","Data":"bba384abc1f8a40f2640005fb8d22b66b17ea8d8730f2b5f2524e8fe20665ba3"}
Sep 17 15:38:59.045217 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:59.045162    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-phv6p" podStartSLOduration=1.314281182 podStartE2EDuration="20.045141768s" totalImagesPullingTime="18.730860586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:59.026957428 +0000 UTC m=+20.917817973" watchObservedRunningTime="2026-09-17 15:38:59.045141768 +0000 UTC m=+20.936002312"
Sep 17 15:38:59.079363 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:59.079191    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-243.ec2.internal" podStartSLOduration=21.079169058 podStartE2EDuration="21.079169058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:59.045089745 +0000 UTC m=+20.935950289" watchObservedRunningTime="2026-09-17 15:38:59.079169058 +0000 UTC m=+20.970029602"
Sep 17 15:38:59.079503 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:59.079418    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gxshr" podStartSLOduration=1.294824276 podStartE2EDuration="20.079411165s" totalImagesPullingTime="18.784586889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:59.073232845 +0000 UTC m=+20.964093387" watchObservedRunningTime="2026-09-17 15:38:59.079411165 +0000 UTC m=+20.970271712"
Sep 17 15:38:59.689316 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:59.689127    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:38:59.689476 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:38:59.689140    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:38:59.689476 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:59.689378    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:38:59.689588 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:38:59.689484    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:39:00.018885 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.018854    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:39:00.019647 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.019246    2555 generic.go:356] "Generic (PLEG): container finished" podID="ab9584e9-4c5a-4603-af6b-e26845d8bf04" containerID="83a0bdfcf6ea62f687bf26bc4fbc955c6fe7731aa0a55998592a76cd2eaedb64" exitCode=1
Sep 17 15:39:00.019647 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.019328    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" event={"ID":"ab9584e9-4c5a-4603-af6b-e26845d8bf04","Type":"ContainerStarted","Data":"395ed38939446f5833cee2bce03a5bc48b40f7327fe9af6395fbcdfad1933300"}
Sep 17 15:39:00.019647 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.019461    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" event={"ID":"ab9584e9-4c5a-4603-af6b-e26845d8bf04","Type":"ContainerStarted","Data":"be4a1e51aad7b4e4b7e2ca09968d7b88f787e4ae33b2627916ae4127655dfa4a"}
Sep 17 15:39:00.019647 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.019509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" event={"ID":"ab9584e9-4c5a-4603-af6b-e26845d8bf04","Type":"ContainerStarted","Data":"dbc41454c123650f5697cca6089ea460f9c2febb9891f95fff5bb57b616d1935"}
Sep 17 15:39:00.019647 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.019552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" event={"ID":"ab9584e9-4c5a-4603-af6b-e26845d8bf04","Type":"ContainerStarted","Data":"b2cffd6e86a9e310647998d3240abcb2f0267343593774d2407b9399f97b0121"}
Sep 17 15:39:00.019647 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.019595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" event={"ID":"ab9584e9-4c5a-4603-af6b-e26845d8bf04","Type":"ContainerDied","Data":"83a0bdfcf6ea62f687bf26bc4fbc955c6fe7731aa0a55998592a76cd2eaedb64"}
Sep 17 15:39:00.019647 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.019638    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" event={"ID":"ab9584e9-4c5a-4603-af6b-e26845d8bf04","Type":"ContainerStarted","Data":"53c9ad77885830516912648548f6905d547c4867c81adf18366759b1228a9bd6"}
Sep 17 15:39:00.021059 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.020987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rjjjt" event={"ID":"6d538ca1-32a3-4bd3-95f4-44177e3d30ab","Type":"ContainerStarted","Data":"255b6c7c41dabb235e390c2ca16ee2fcc06689538ca16a48d05bc6ec4441e153"}
Sep 17 15:39:00.022730 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.022670    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss" event={"ID":"6600b1ea-0e03-4858-9e42-77e0779cd3bd","Type":"ContainerStarted","Data":"2519aa762e7e7ba937c716b44c2920f4ca4796e86531c1598bbcf752b3857f32"}
Sep 17 15:39:00.024267 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.024241    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4k758" event={"ID":"33b10e70-61bd-4101-9a26-e0349b249102","Type":"ContainerStarted","Data":"08820e901357a75808bafe2a40b438fe6c424fadaf7b0cca36ff154437800a22"}
Sep 17 15:39:00.026086 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.026054    2555 generic.go:356] "Generic (PLEG): container finished" podID="eff4a4436cf19c431e9ef9db87e16884" containerID="0bb7e2d77f4aebc6996e242d2a2104cea16bd12dbbefd6f82e2839aed4718a09" exitCode=0
Sep 17 15:39:00.026173 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.026145    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal" event={"ID":"eff4a4436cf19c431e9ef9db87e16884","Type":"ContainerDied","Data":"0bb7e2d77f4aebc6996e242d2a2104cea16bd12dbbefd6f82e2839aed4718a09"}
Sep 17 15:39:00.027842 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.027810    2555 generic.go:356] "Generic (PLEG): container finished" podID="c611bed6-3ca1-412d-9047-b574034b42d8" containerID="f2291f8da657b20fd024b2cce163f732111f1fb9b181ab7a302da8e9b3b4fe30" exitCode=0
Sep 17 15:39:00.027929 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.027850    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjwg8" event={"ID":"c611bed6-3ca1-412d-9047-b574034b42d8","Type":"ContainerDied","Data":"f2291f8da657b20fd024b2cce163f732111f1fb9b181ab7a302da8e9b3b4fe30"}
Sep 17 15:39:00.029492 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.029473    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p65h2" event={"ID":"fd89a8d5-387c-4f46-80d2-1c7ad2bc8286","Type":"ContainerStarted","Data":"cff74fd4f6281652c7871de683ba22296bc6c16686c076320b2ebeaafc41663c"}
Sep 17 15:39:00.037057 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.037019    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rjjjt" podStartSLOduration=3.298450051 podStartE2EDuration="22.037007005s" totalImagesPullingTime="18.738556954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:00.036702596 +0000 UTC m=+21.927563139" watchObservedRunningTime="2026-09-17 15:39:00.037007005 +0000 UTC m=+21.927867551"
Sep 17 15:39:00.051120 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.051074    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4k758" podStartSLOduration=3.24639518 podStartE2EDuration="22.051060434s" totalImagesPullingTime="18.804665254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:00.050480077 +0000 UTC m=+21.941340621" watchObservedRunningTime="2026-09-17 15:39:00.051060434 +0000 UTC m=+21.941920978"
Sep 17 15:39:00.065212 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.065169    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p65h2" podStartSLOduration=9.925940145 podStartE2EDuration="21.065154828s" totalImagesPullingTime="11.139214683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:00.064874698 +0000 UTC m=+21.955735242" watchObservedRunningTime="2026-09-17 15:39:00.065154828 +0000 UTC m=+21.956015372"
Sep 17 15:39:00.412753 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.412590    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:39:00.654957 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.654796    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:39:00.412749712Z","UUID":"dc1a35fb-f127-46b5-9f73-732bd476fb9c","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:39:00.656142 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.656116    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:39:00.656142 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.656143    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:39:00.688590 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:00.688564    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:39:00.688742 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:00.688688    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:39:01.033991 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:01.033910    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal" event={"ID":"eff4a4436cf19c431e9ef9db87e16884","Type":"ContainerStarted","Data":"975ffd89e48a9ebd9b84d9e67f5415e80df7646807b4faa5df7dd75715d8c11c"}
Sep 17 15:39:01.035843 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:01.035797    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vqm4m" event={"ID":"c1ee4a7c-2b79-4c3f-9405-c67719f47c54","Type":"ContainerStarted","Data":"8e378f28be8f3b99081c18f62b8e6005f58a3d4bb136c562188c4068d21b340b"}
Sep 17 15:39:01.038274 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:01.038242    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss" event={"ID":"6600b1ea-0e03-4858-9e42-77e0779cd3bd","Type":"ContainerStarted","Data":"3ace1dbc14ae5969bde651ebf5a98321a51fcfc6bbce1b0845c9a34f65215b05"}
Sep 17 15:39:01.056139 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:01.056090    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-243.ec2.internal" podStartSLOduration=23.056076807 podStartE2EDuration="23.056076807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:01.05593826 +0000 UTC m=+22.946798806" watchObservedRunningTime="2026-09-17 15:39:01.056076807 +0000 UTC m=+22.946937352"
Sep 17 15:39:01.076492 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:01.076339    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vqm4m" podStartSLOduration=22.076322376 podStartE2EDuration="22.076322376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-17 15:39:01.075512187 +0000 UTC m=+22.966372732" watchObservedRunningTime="2026-09-17 15:39:01.076322376 +0000 UTC m=+22.967182922"
Sep 17 15:39:01.688679 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:01.688652    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:01.688956 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:01.688681    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:01.688956 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:01.688902    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:39:01.688956 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:01.688897    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:39:02.043111 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:02.043044    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:39:02.043476 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:02.043456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" event={"ID":"ab9584e9-4c5a-4603-af6b-e26845d8bf04","Type":"ContainerStarted","Data":"ca085e1595a1217990a388a0c7f43f23a3ec77c37990a262a241e89df9dea1cc"}
Sep 17 15:39:02.045364 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:02.045342    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss" event={"ID":"6600b1ea-0e03-4858-9e42-77e0779cd3bd","Type":"ContainerStarted","Data":"fceade1b22f884b4032d41aabdae5f1a0747a01d888b9e26000b72849b9389c2"}
Sep 17 15:39:02.067890 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:02.067838    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9x9ss" podStartSLOduration=3.333401362 podStartE2EDuration="24.067803328s" totalImagesPullingTime="20.734401966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:02.067142634 +0000 UTC m=+23.958003178" watchObservedRunningTime="2026-09-17 15:39:02.067803328 +0000 UTC m=+23.958663875"
Sep 17 15:39:02.689079 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:02.689047    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:39:02.689258 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:02.689172    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:39:02.728040 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:02.728009    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:39:02.728688 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:02.728632    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:39:03.047955 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:03.047869    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:39:03.048367 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:03.048347    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p65h2"
Sep 17 15:39:03.688364 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:03.688332    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:03.688557 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:03.688372    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:03.688557 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:03.688456    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:39:03.688689 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:03.688567    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:39:04.688890 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:04.688868    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:39:04.689278 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:04.688975    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:39:05.053004 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:05.052785    2555 generic.go:356] "Generic (PLEG): container finished" podID="c611bed6-3ca1-412d-9047-b574034b42d8" containerID="e105da528555d4fc1ac4adc8835e893984a842e6d76876273899790c94f642d9" exitCode=0
Sep 17 15:39:05.053129 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:05.052855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjwg8" event={"ID":"c611bed6-3ca1-412d-9047-b574034b42d8","Type":"ContainerDied","Data":"e105da528555d4fc1ac4adc8835e893984a842e6d76876273899790c94f642d9"}
Sep 17 15:39:05.056944 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:05.056928    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:39:05.057356 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:05.057337    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" event={"ID":"ab9584e9-4c5a-4603-af6b-e26845d8bf04","Type":"ContainerStarted","Data":"8994cd75be125fd9c4901606e6f0ceaea56ff8d922b15c386c5d29599b630636"}
Sep 17 15:39:05.057566 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:05.057551    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:39:05.057733 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:05.057718    2555 scope.go:117] "RemoveContainer" containerID="83a0bdfcf6ea62f687bf26bc4fbc955c6fe7731aa0a55998592a76cd2eaedb64"
Sep 17 15:39:05.688376 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:05.688347    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:05.688543 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:05.688459    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:39:05.688543 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:05.688510    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:05.688615 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:05.688597    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:39:06.063360 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.063293    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:39:06.063816 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.063696    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" event={"ID":"ab9584e9-4c5a-4603-af6b-e26845d8bf04","Type":"ContainerStarted","Data":"2de92cd535f09a894489927cf209b168b5c4513077166b041c6c7cbc301da186"}
Sep 17 15:39:06.063994 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.063975    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:39:06.064073 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.064056    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:39:06.066257 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.066227    2555 generic.go:356] "Generic (PLEG): container finished" podID="c611bed6-3ca1-412d-9047-b574034b42d8" containerID="32fb098d50fde3b42c6e82911e2b49892bef6603b4d50d2e62abbcc1b7b30292" exitCode=0
Sep 17 15:39:06.066257 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.066257    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjwg8" event={"ID":"c611bed6-3ca1-412d-9047-b574034b42d8","Type":"ContainerDied","Data":"32fb098d50fde3b42c6e82911e2b49892bef6603b4d50d2e62abbcc1b7b30292"}
Sep 17 15:39:06.079933 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.079913    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:39:06.080024 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.080013    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:39:06.093818 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.093779    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx" podStartSLOduration=8.1329694 podStartE2EDuration="27.093766471s" totalImagesPullingTime="18.960797071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:06.091909718 +0000 UTC m=+27.982770263" watchObservedRunningTime="2026-09-17 15:39:06.093766471 +0000 UTC m=+27.984627015"
Sep 17 15:39:06.639949 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.639910    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6nj4l"]
Sep 17 15:39:06.640109 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.640027    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:06.640168 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:06.640118    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:39:06.643261 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.643217    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b2879"]
Sep 17 15:39:06.643384 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.643343    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:39:06.643479 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:06.643457    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:39:06.644045 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.644021    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nlf6d"]
Sep 17 15:39:06.644148 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:06.644110    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:06.644208 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:06.644189    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:39:07.073107 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:07.071092    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjwg8" event={"ID":"c611bed6-3ca1-412d-9047-b574034b42d8","Type":"ContainerDied","Data":"d057c9c1d6bdd624471dfb851977b3331d072dd293fd713f767d497727b96565"}
Sep 17 15:39:07.073107 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:07.071058    2555 generic.go:356] "Generic (PLEG): container finished" podID="c611bed6-3ca1-412d-9047-b574034b42d8" containerID="d057c9c1d6bdd624471dfb851977b3331d072dd293fd713f767d497727b96565" exitCode=0
Sep 17 15:39:07.688973 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:07.688948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:07.689125 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:07.689045    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:39:08.689784 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:08.689566    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:39:08.690341 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:08.689648    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:08.690341 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:08.689857    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:39:08.690341 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:08.689896    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:39:09.688678 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:09.688641    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:09.688874 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:09.688766    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:39:10.688769 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:10.688727    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:39:10.689267 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:10.688750    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:10.689267 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:10.688891    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b2879" podUID="2cf8474f-71ab-45db-bfaa-e4953f520025"
Sep 17 15:39:10.689267 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:10.688953    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nlf6d" podUID="a6acb82a-8eb6-4508-9345-88b203e27b0e"
Sep 17 15:39:11.343491 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:11.343458    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:39:11.343664 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:11.343556    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:11.343664 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:11.343605    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs podName:2cf8474f-71ab-45db-bfaa-e4953f520025 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:43.343591384 +0000 UTC m=+65.234451906 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs") pod "network-metrics-daemon-b2879" (UID: "2cf8474f-71ab-45db-bfaa-e4953f520025") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:11.444663 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:11.444633    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz\") pod \"network-check-target-6nj4l\" (UID: \"01547a09-a86f-4c51-989e-e4d46ad48c5f\") " pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:11.444765 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:11.444676    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:11.444804 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:11.444772    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:11.444804 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:11.444787    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:11.444886 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:11.444815    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:11.444886 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:11.444842    2555 projected.go:196] Error preparing data for projected volume kube-api-access-nv6sz for pod openshift-network-diagnostics/network-check-target-6nj4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:11.444886 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:11.444850    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret podName:a6acb82a-8eb6-4508-9345-88b203e27b0e nodeName:}" failed. No retries permitted until 2026-09-17 15:39:43.444819702 +0000 UTC m=+65.335680239 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret") pod "global-pull-secret-syncer-nlf6d" (UID: "a6acb82a-8eb6-4508-9345-88b203e27b0e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:11.444886 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:11.444879    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz podName:01547a09-a86f-4c51-989e-e4d46ad48c5f nodeName:}" failed. No retries permitted until 2026-09-17 15:39:43.44486797 +0000 UTC m=+65.335728497 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv6sz" (UniqueName: "kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz") pod "network-check-target-6nj4l" (UID: "01547a09-a86f-4c51-989e-e4d46ad48c5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:11.688583 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:11.688554    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:11.688777 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:11.688650    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6nj4l" podUID="01547a09-a86f-4c51-989e-e4d46ad48c5f"
Sep 17 15:39:11.939769 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:11.939741    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-243.ec2.internal" event="NodeReady"
Sep 17 15:39:11.940366 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:11.939897    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:39:11.976074 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:11.976051    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64b78c745c-47ztw"]
Sep 17 15:39:11.998644 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:11.998623    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p2dt6"]
Sep 17 15:39:11.998757 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:11.998739    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.001613 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.001592    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 15:39:12.002156 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.002135    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w2hrw\""
Sep 17 15:39:12.002500 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.002481    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 15:39:12.002591 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.002543    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 15:39:12.010559 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.010536    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 15:39:12.014518 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.014498    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64b78c745c-47ztw"]
Sep 17 15:39:12.014608 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.014534    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h5tw5"]
Sep 17 15:39:12.014652 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.014606    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.018467 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.018429    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9xnvp\""
Sep 17 15:39:12.018575 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.018558    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:39:12.019032 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.019010    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:39:12.032569 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.032552    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h5tw5"]
Sep 17 15:39:12.032656 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.032577    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p2dt6"]
Sep 17 15:39:12.032656 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.032639    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h5tw5"
Sep 17 15:39:12.035255 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.035237    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:39:12.035515 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.035499    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:39:12.035595 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.035502    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:39:12.035783 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.035769    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lggbn\""
Sep 17 15:39:12.087141 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.087118    2555 generic.go:356] "Generic (PLEG): container finished" podID="c611bed6-3ca1-412d-9047-b574034b42d8" containerID="63b69789ef1a545cd1152d4aa59ea359a98975648ce28394265303e24c50e039" exitCode=0
Sep 17 15:39:12.087225 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.087176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjwg8" event={"ID":"c611bed6-3ca1-412d-9047-b574034b42d8","Type":"ContainerDied","Data":"63b69789ef1a545cd1152d4aa59ea359a98975648ce28394265303e24c50e039"}
Sep 17 15:39:12.150969 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.150944    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-image-registry-private-configuration\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.151073 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.150982    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-installation-pull-secrets\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.151073 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151014    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-trusted-ca\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.151073 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-bound-sa-token\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.151225 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151133    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tjcm\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-kube-api-access-8tjcm\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.151271 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151214    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/914dd600-d7ac-49f2-b490-5a4d82d748ae-config-volume\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.151271 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151249    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5zj\" (UniqueName: \"kubernetes.io/projected/914dd600-d7ac-49f2-b490-5a4d82d748ae-kube-api-access-mm5zj\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.151354 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151321    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.151354 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151347    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.151442 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151377    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3f3fb6-88e6-4198-b631-05b2a0f548cc-ca-trust-extracted\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.151442 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151404    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9addd732-ac2d-4f71-9057-def9b3c7fbe5-cert\") pod \"ingress-canary-h5tw5\" (UID: \"9addd732-ac2d-4f71-9057-def9b3c7fbe5\") " pod="openshift-ingress-canary/ingress-canary-h5tw5"
Sep 17 15:39:12.151533 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151443    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914dd600-d7ac-49f2-b490-5a4d82d748ae-tmp-dir\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.151533 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151469    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtz6\" (UniqueName: \"kubernetes.io/projected/9addd732-ac2d-4f71-9057-def9b3c7fbe5-kube-api-access-gbtz6\") pod \"ingress-canary-h5tw5\" (UID: \"9addd732-ac2d-4f71-9057-def9b3c7fbe5\") " pod="openshift-ingress-canary/ingress-canary-h5tw5"
Sep 17 15:39:12.151533 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.151492    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-certificates\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.183972 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.183948    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p65h2_fd89a8d5-387c-4f46-80d2-1c7ad2bc8286/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:12.252661 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252633    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9addd732-ac2d-4f71-9057-def9b3c7fbe5-cert\") pod \"ingress-canary-h5tw5\" (UID: \"9addd732-ac2d-4f71-9057-def9b3c7fbe5\") " pod="openshift-ingress-canary/ingress-canary-h5tw5"
Sep 17 15:39:12.252757 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252675    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914dd600-d7ac-49f2-b490-5a4d82d748ae-tmp-dir\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.252757 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtz6\" (UniqueName: \"kubernetes.io/projected/9addd732-ac2d-4f71-9057-def9b3c7fbe5-kube-api-access-gbtz6\") pod \"ingress-canary-h5tw5\" (UID: \"9addd732-ac2d-4f71-9057-def9b3c7fbe5\") " pod="openshift-ingress-canary/ingress-canary-h5tw5"
Sep 17 15:39:12.252757 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252721    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-certificates\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.252757 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252752    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-image-registry-private-configuration\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.252971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252772    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-installation-pull-secrets\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.252971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252800    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-trusted-ca\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.252971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252841    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-bound-sa-token\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.252971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tjcm\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-kube-api-access-8tjcm\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.252971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252926    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/914dd600-d7ac-49f2-b490-5a4d82d748ae-config-volume\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.252971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5zj\" (UniqueName: \"kubernetes.io/projected/914dd600-d7ac-49f2-b490-5a4d82d748ae-kube-api-access-mm5zj\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.253268 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.252992    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.253268 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.253017    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.253268 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.253048    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3f3fb6-88e6-4198-b631-05b2a0f548cc-ca-trust-extracted\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.253268 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.253176    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914dd600-d7ac-49f2-b490-5a4d82d748ae-tmp-dir\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.253471 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:12.253284    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:12.253471 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:12.253342    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls podName:914dd600-d7ac-49f2-b490-5a4d82d748ae nodeName:}" failed. No retries permitted until 2026-09-17 15:39:12.753322958 +0000 UTC m=+34.644183484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls") pod "dns-default-p2dt6" (UID: "914dd600-d7ac-49f2-b490-5a4d82d748ae") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:12.253471 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.253371    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-certificates\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.253625 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:12.253507    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:12.253625 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:12.253522    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b78c745c-47ztw: secret "image-registry-tls" not found
Sep 17 15:39:12.253625 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:12.253568    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls podName:da3f3fb6-88e6-4198-b631-05b2a0f548cc nodeName:}" failed. No retries permitted until 2026-09-17 15:39:12.753549786 +0000 UTC m=+34.644410307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls") pod "image-registry-64b78c745c-47ztw" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc") : secret "image-registry-tls" not found
Sep 17 15:39:12.253841 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.253802    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/914dd600-d7ac-49f2-b490-5a4d82d748ae-config-volume\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.253912 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.253849    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3f3fb6-88e6-4198-b631-05b2a0f548cc-ca-trust-extracted\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.254168 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.254120    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-trusted-ca\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.256955 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.256933    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-image-registry-private-configuration\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.256955 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.256943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9addd732-ac2d-4f71-9057-def9b3c7fbe5-cert\") pod \"ingress-canary-h5tw5\" (UID: \"9addd732-ac2d-4f71-9057-def9b3c7fbe5\") " pod="openshift-ingress-canary/ingress-canary-h5tw5"
Sep 17 15:39:12.257073 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.256940    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-installation-pull-secrets\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.262869 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.262803    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-bound-sa-token\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.263224 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.263209    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tjcm\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-kube-api-access-8tjcm\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.263373 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.263353    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtz6\" (UniqueName: \"kubernetes.io/projected/9addd732-ac2d-4f71-9057-def9b3c7fbe5-kube-api-access-gbtz6\") pod \"ingress-canary-h5tw5\" (UID: \"9addd732-ac2d-4f71-9057-def9b3c7fbe5\") " pod="openshift-ingress-canary/ingress-canary-h5tw5"
Sep 17 15:39:12.263412 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.263353    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5zj\" (UniqueName: \"kubernetes.io/projected/914dd600-d7ac-49f2-b490-5a4d82d748ae-kube-api-access-mm5zj\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.340638 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.340617    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h5tw5"
Sep 17 15:39:12.645057 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.645031    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h5tw5"]
Sep 17 15:39:12.648690 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:39:12.648661    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9addd732_ac2d_4f71_9057_def9b3c7fbe5.slice/crio-f74646f5230c26f089d300755fdd00394ce4fe4e3450a11dca63daef1c909167 WatchSource:0}: Error finding container f74646f5230c26f089d300755fdd00394ce4fe4e3450a11dca63daef1c909167: Status 404 returned error can't find the container with id f74646f5230c26f089d300755fdd00394ce4fe4e3450a11dca63daef1c909167
Sep 17 15:39:12.688970 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.688946    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:39:12.689075 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.688946    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:12.692024 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.692006    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:39:12.692255 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.692235    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:39:12.692335 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.692296    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j5497\""
Sep 17 15:39:12.757101 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.757082    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:12.757188 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:12.757105    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:12.757236 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:12.757209    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:12.757279 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:12.757248    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:12.757279 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:12.757257    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b78c745c-47ztw: secret "image-registry-tls" not found
Sep 17 15:39:12.757279 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:12.757261    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls podName:914dd600-d7ac-49f2-b490-5a4d82d748ae nodeName:}" failed. No retries permitted until 2026-09-17 15:39:13.757246209 +0000 UTC m=+35.648106746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls") pod "dns-default-p2dt6" (UID: "914dd600-d7ac-49f2-b490-5a4d82d748ae") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:12.757385 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:12.757290    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls podName:da3f3fb6-88e6-4198-b631-05b2a0f548cc nodeName:}" failed. No retries permitted until 2026-09-17 15:39:13.757279074 +0000 UTC m=+35.648139597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls") pod "image-registry-64b78c745c-47ztw" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc") : secret "image-registry-tls" not found
Sep 17 15:39:13.090339 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:13.090302    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h5tw5" event={"ID":"9addd732-ac2d-4f71-9057-def9b3c7fbe5","Type":"ContainerStarted","Data":"f74646f5230c26f089d300755fdd00394ce4fe4e3450a11dca63daef1c909167"}
Sep 17 15:39:13.093515 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:13.093488    2555 generic.go:356] "Generic (PLEG): container finished" podID="c611bed6-3ca1-412d-9047-b574034b42d8" containerID="8a14e3dc8d577fb17daabe018a37e17d661ad6b7b37c3e93d4526dc7a6791d36" exitCode=0
Sep 17 15:39:13.093649 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:13.093529    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjwg8" event={"ID":"c611bed6-3ca1-412d-9047-b574034b42d8","Type":"ContainerDied","Data":"8a14e3dc8d577fb17daabe018a37e17d661ad6b7b37c3e93d4526dc7a6791d36"}
Sep 17 15:39:13.413186 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:13.413080    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p65h2_fd89a8d5-387c-4f46-80d2-1c7ad2bc8286/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:13.688644 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:13.688568    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:13.692674 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:13.692649    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xkd6d\""
Sep 17 15:39:13.692794 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:13.692693    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:39:13.692794 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:13.692653    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:39:13.768052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:13.768023    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:13.768052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:13.768053    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:13.768259 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:13.768192    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:13.768259 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:13.768254    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls podName:914dd600-d7ac-49f2-b490-5a4d82d748ae nodeName:}" failed. No retries permitted until 2026-09-17 15:39:15.768238587 +0000 UTC m=+37.659099113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls") pod "dns-default-p2dt6" (UID: "914dd600-d7ac-49f2-b490-5a4d82d748ae") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:13.768369 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:13.768196    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:13.768369 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:13.768299    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b78c745c-47ztw: secret "image-registry-tls" not found
Sep 17 15:39:13.768369 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:13.768357    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls podName:da3f3fb6-88e6-4198-b631-05b2a0f548cc nodeName:}" failed. No retries permitted until 2026-09-17 15:39:15.768343625 +0000 UTC m=+37.659204161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls") pod "image-registry-64b78c745c-47ztw" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc") : secret "image-registry-tls" not found
Sep 17 15:39:14.099881 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:14.099849    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pjwg8" event={"ID":"c611bed6-3ca1-412d-9047-b574034b42d8","Type":"ContainerStarted","Data":"b313c3f1252fd781da0939bd6409ca82502d5ec7e4eae52905da16114ee1b4c2"}
Sep 17 15:39:14.124606 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:14.124562    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pjwg8" podStartSLOduration=6.133710096 podStartE2EDuration="35.12454842s" totalImagesPullingTime="28.990838324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:14.124377736 +0000 UTC m=+36.015238301" watchObservedRunningTime="2026-09-17 15:39:14.12454842 +0000 UTC m=+36.015408955"
Sep 17 15:39:14.244926 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:14.244890    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p65h2_fd89a8d5-387c-4f46-80d2-1c7ad2bc8286/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:14.304612 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:14.304580    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p65h2_fd89a8d5-387c-4f46-80d2-1c7ad2bc8286/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:15.103979 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:15.103951    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h5tw5" event={"ID":"9addd732-ac2d-4f71-9057-def9b3c7fbe5","Type":"ContainerStarted","Data":"2e42657720205003b016fe38e53eb6f24d1ec4f1107b76932aa22e933cdb06b9"}
Sep 17 15:39:15.124581 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:15.124539    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h5tw5" podStartSLOduration=1.782921783 podStartE2EDuration="4.124523811s" totalImagesPullingTime="2.341602028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:15.124102372 +0000 UTC m=+37.014962916" watchObservedRunningTime="2026-09-17 15:39:15.124523811 +0000 UTC m=+37.015384354"
Sep 17 15:39:15.765094 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:15.765051    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p65h2_fd89a8d5-387c-4f46-80d2-1c7ad2bc8286/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:15.782957 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:15.782932    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:15.783036 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:15.782963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:15.783073 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:15.783063    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:15.783106 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:15.783077    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b78c745c-47ztw: secret "image-registry-tls" not found
Sep 17 15:39:15.783139 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:15.783117    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls podName:da3f3fb6-88e6-4198-b631-05b2a0f548cc nodeName:}" failed. No retries permitted until 2026-09-17 15:39:19.783105222 +0000 UTC m=+41.673965744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls") pod "image-registry-64b78c745c-47ztw" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc") : secret "image-registry-tls" not found
Sep 17 15:39:15.783182 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:15.783065    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:15.783213 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:15.783205    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls podName:914dd600-d7ac-49f2-b490-5a4d82d748ae nodeName:}" failed. No retries permitted until 2026-09-17 15:39:19.783190024 +0000 UTC m=+41.674050551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls") pod "dns-default-p2dt6" (UID: "914dd600-d7ac-49f2-b490-5a4d82d748ae") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:15.864455 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:15.864431    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p65h2_fd89a8d5-387c-4f46-80d2-1c7ad2bc8286/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:16.381959 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:16.381931    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p65h2_fd89a8d5-387c-4f46-80d2-1c7ad2bc8286/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:16.484234 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:16.484158    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p65h2_fd89a8d5-387c-4f46-80d2-1c7ad2bc8286/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:17.360780 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:17.360751    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p65h2_fd89a8d5-387c-4f46-80d2-1c7ad2bc8286/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:19.811735 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:19.811705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:19.811735 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:19.811736    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:19.812137 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:19.811851    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:19.812137 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:19.811911    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls podName:914dd600-d7ac-49f2-b490-5a4d82d748ae nodeName:}" failed. No retries permitted until 2026-09-17 15:39:27.811896946 +0000 UTC m=+49.702757468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls") pod "dns-default-p2dt6" (UID: "914dd600-d7ac-49f2-b490-5a4d82d748ae") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:19.812137 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:19.811851    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:19.812137 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:19.811932    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b78c745c-47ztw: secret "image-registry-tls" not found
Sep 17 15:39:19.812137 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:19.811953    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls podName:da3f3fb6-88e6-4198-b631-05b2a0f548cc nodeName:}" failed. No retries permitted until 2026-09-17 15:39:27.81194727 +0000 UTC m=+49.702807791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls") pod "image-registry-64b78c745c-47ztw" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc") : secret "image-registry-tls" not found
Sep 17 15:39:27.867349 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:27.867306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:27.867349 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:27.867348    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:27.867755 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:27.867453    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:27.867755 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:27.867465    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b78c745c-47ztw: secret "image-registry-tls" not found
Sep 17 15:39:27.867755 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:27.867472    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:27.867755 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:27.867524    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls podName:da3f3fb6-88e6-4198-b631-05b2a0f548cc nodeName:}" failed. No retries permitted until 2026-09-17 15:39:43.86751135 +0000 UTC m=+65.758371872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls") pod "image-registry-64b78c745c-47ztw" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc") : secret "image-registry-tls" not found
Sep 17 15:39:27.867755 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:27.867536    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls podName:914dd600-d7ac-49f2-b490-5a4d82d748ae nodeName:}" failed. No retries permitted until 2026-09-17 15:39:43.867530948 +0000 UTC m=+65.758391469 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls") pod "dns-default-p2dt6" (UID: "914dd600-d7ac-49f2-b490-5a4d82d748ae") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:38.084607 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:38.084579    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7dwx"
Sep 17 15:39:43.374589 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.374556    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:39:43.377658 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.377640    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:39:43.385113 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:43.385098    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 15:39:43.385155 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:43.385148    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs podName:2cf8474f-71ab-45db-bfaa-e4953f520025 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:47.38513344 +0000 UTC m=+129.275993975 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs") pod "network-metrics-daemon-b2879" (UID: "2cf8474f-71ab-45db-bfaa-e4953f520025") : secret "metrics-daemon-secret" not found
Sep 17 15:39:43.475345 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.475322    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz\") pod \"network-check-target-6nj4l\" (UID: \"01547a09-a86f-4c51-989e-e4d46ad48c5f\") " pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:43.475428 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.475353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:43.478206 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.478184    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:39:43.478315 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.478237    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:39:43.488350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.488331    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:39:43.488598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.488582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6acb82a-8eb6-4508-9345-88b203e27b0e-original-pull-secret\") pod \"global-pull-secret-syncer-nlf6d\" (UID: \"a6acb82a-8eb6-4508-9345-88b203e27b0e\") " pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:43.498561 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.498534    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/01547a09-a86f-4c51-989e-e4d46ad48c5f-kube-api-access-nv6sz\") pod \"network-check-target-6nj4l\" (UID: \"01547a09-a86f-4c51-989e-e4d46ad48c5f\") " pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:43.603949 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.603930    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nlf6d"
Sep 17 15:39:43.701060 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.701031    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xkd6d\""
Sep 17 15:39:43.708760 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.708742    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:43.875694 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.875662    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nlf6d"]
Sep 17 15:39:43.878575 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.878546    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:39:43.878754 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.878584    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:39:43.878754 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:39:43.878636    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6acb82a_8eb6_4508_9345_88b203e27b0e.slice/crio-67a2d14db73d05c1f3e80618ae05d640e2a627df090b554a2749c23f747d81a1 WatchSource:0}: Error finding container 67a2d14db73d05c1f3e80618ae05d640e2a627df090b554a2749c23f747d81a1: Status 404 returned error can't find the container with id 67a2d14db73d05c1f3e80618ae05d640e2a627df090b554a2749c23f747d81a1
Sep 17 15:39:43.878754 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:43.878693    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:43.878754 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:43.878697    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:43.878754 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:43.878704    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b78c745c-47ztw: secret "image-registry-tls" not found
Sep 17 15:39:43.878754 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:43.878754    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls podName:da3f3fb6-88e6-4198-b631-05b2a0f548cc nodeName:}" failed. No retries permitted until 2026-09-17 15:40:15.878741702 +0000 UTC m=+97.769602224 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls") pod "image-registry-64b78c745c-47ztw" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc") : secret "image-registry-tls" not found
Sep 17 15:39:43.879035 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:43.878772    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls podName:914dd600-d7ac-49f2-b490-5a4d82d748ae nodeName:}" failed. No retries permitted until 2026-09-17 15:40:15.878762773 +0000 UTC m=+97.769623296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls") pod "dns-default-p2dt6" (UID: "914dd600-d7ac-49f2-b490-5a4d82d748ae") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:43.971566 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:43.971518    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6nj4l"]
Sep 17 15:39:43.974035 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:39:43.974015    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01547a09_a86f_4c51_989e_e4d46ad48c5f.slice/crio-cc229833df8fb87b69933e61afa328a88c5edc9ccc0c0b95eb2d6420e87658fa WatchSource:0}: Error finding container cc229833df8fb87b69933e61afa328a88c5edc9ccc0c0b95eb2d6420e87658fa: Status 404 returned error can't find the container with id cc229833df8fb87b69933e61afa328a88c5edc9ccc0c0b95eb2d6420e87658fa
Sep 17 15:39:44.169213 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:44.169183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6nj4l" event={"ID":"01547a09-a86f-4c51-989e-e4d46ad48c5f","Type":"ContainerStarted","Data":"cc229833df8fb87b69933e61afa328a88c5edc9ccc0c0b95eb2d6420e87658fa"}
Sep 17 15:39:44.170188 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:44.170167    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nlf6d" event={"ID":"a6acb82a-8eb6-4508-9345-88b203e27b0e","Type":"ContainerStarted","Data":"67a2d14db73d05c1f3e80618ae05d640e2a627df090b554a2749c23f747d81a1"}
Sep 17 15:39:49.185801 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:49.185759    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6nj4l" event={"ID":"01547a09-a86f-4c51-989e-e4d46ad48c5f","Type":"ContainerStarted","Data":"8cf2d0a9754cb65810e6f45242dbb5b22032be465e79c924cd2daf151f4853bd"}
Sep 17 15:39:49.186261 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:49.185885    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:39:49.187258 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:49.187232    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nlf6d" event={"ID":"a6acb82a-8eb6-4508-9345-88b203e27b0e","Type":"ContainerStarted","Data":"7df81e44c4d603eda2d2c52bfcc3635bcefc76199a69fb1b876d7087bef9372d"}
Sep 17 15:39:49.201878 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:49.201817    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6nj4l" podStartSLOduration=66.89920195 podStartE2EDuration="1m11.201803882s" totalImagesPullingTime="4.302601932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:49.200490113 +0000 UTC m=+71.091350658" watchObservedRunningTime="2026-09-17 15:39:49.201803882 +0000 UTC m=+71.092664450"
Sep 17 15:39:49.213719 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:49.213675    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nlf6d" podStartSLOduration=65.813107856 podStartE2EDuration="1m10.213660027s" totalImagesPullingTime="4.400552171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:49.213606051 +0000 UTC m=+71.104466596" watchObservedRunningTime="2026-09-17 15:39:49.213660027 +0000 UTC m=+71.104520571"
Sep 17 15:39:52.095452 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.095416    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"]
Sep 17 15:39:52.097373 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.097356    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:39:52.101227 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.101206    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 15:39:52.101534 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.101516    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-br9nh\""
Sep 17 15:39:52.101627 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.101529    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:39:52.102436 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.102419    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 15:39:52.112100 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.112079    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"]
Sep 17 15:39:52.208684 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.208652    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"]
Sep 17 15:39:52.210519 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.210497    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:52.213071 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:52.213045    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 17 15:39:52.213071 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:52.213048    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"cluster-monitoring-operator-6cd954b897-cmkxg\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" podUID="1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:52.213410 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:52.213385    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-monitoring-operator-tls\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret"
Sep 17 15:39:52.213990 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:52.213967    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"telemetry-config\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap"
Sep 17 15:39:52.214053 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:52.213999    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-monitoring-operator-dockercfg-6fktr\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6fktr\"" type="*v1.Secret"
Sep 17 15:39:52.214862 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:52.214842    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 17 15:39:52.230112 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.230088    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"]
Sep 17 15:39:52.241533 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.241509    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:39:52.241637 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.241550    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gbh8\" (UniqueName: \"kubernetes.io/projected/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-kube-api-access-5gbh8\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:39:52.341997 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.341960    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:39:52.342143 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.342001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r972h\" (UniqueName: \"kubernetes.io/projected/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-kube-api-access-r972h\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:52.342143 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.342068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gbh8\" (UniqueName: \"kubernetes.io/projected/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-kube-api-access-5gbh8\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:39:52.342143 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.342126    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:52.342143 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:52.342132    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:39:52.342316 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.342154    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:52.342316 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:52.342210    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls podName:c4eef5bb-eeae-4a79-8362-1b37ca2499f3 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:52.842189 +0000 UTC m=+74.733049535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-n47fv" (UID: "c4eef5bb-eeae-4a79-8362-1b37ca2499f3") : secret "samples-operator-tls" not found
Sep 17 15:39:52.359779 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.359714    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gbh8\" (UniqueName: \"kubernetes.io/projected/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-kube-api-access-5gbh8\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:39:52.442745 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.442709    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:52.442745 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.442750    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:52.442905 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.442805    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r972h\" (UniqueName: \"kubernetes.io/projected/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-kube-api-access-r972h\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:52.844735 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:52.844710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:39:52.844887 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:52.844859    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:39:52.844935 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:52.844920    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls podName:c4eef5bb-eeae-4a79-8362-1b37ca2499f3 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:53.844905422 +0000 UTC m=+75.735765944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-n47fv" (UID: "c4eef5bb-eeae-4a79-8362-1b37ca2499f3") : secret "samples-operator-tls" not found
Sep 17 15:39:53.040661 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:53.040631    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:39:53.157561 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:53.157501    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 15:39:53.164841 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:53.164796    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:53.248006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:53.247980    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 15:39:53.253551 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:53.253536    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:39:53.253630 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:53.253588    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls podName:1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:53.753574185 +0000 UTC m=+75.644434707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cmkxg" (UID: "1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:39:53.343392 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:53.343372    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6fktr\""
Sep 17 15:39:53.449544 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:53.449492    2555 projected.go:291] Couldn't get configMap openshift-monitoring/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 17 15:39:53.449544 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:53.449527    2555 projected.go:196] Error preparing data for projected volume kube-api-access-r972h for pod openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg: failed to sync configmap cache: timed out waiting for the condition
Sep 17 15:39:53.449641 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:53.449584    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-kube-api-access-r972h podName:1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:53.949570896 +0000 UTC m=+75.840431418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r972h" (UniqueName: "kubernetes.io/projected/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-kube-api-access-r972h") pod "cluster-monitoring-operator-6cd954b897-cmkxg" (UID: "1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 15:39:53.753967 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:53.753910    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:39:53.850890 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:53.850856    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:53.850991 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:53.850920    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:39:53.851067 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:53.851042    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:39:53.851120 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:53.851083    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:39:53.851167 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:53.851139    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls podName:1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:54.851104982 +0000 UTC m=+76.741965518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cmkxg" (UID: "1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:39:53.851230 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:53.851174    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls podName:c4eef5bb-eeae-4a79-8362-1b37ca2499f3 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:55.851161827 +0000 UTC m=+77.742022348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-n47fv" (UID: "c4eef5bb-eeae-4a79-8362-1b37ca2499f3") : secret "samples-operator-tls" not found
Sep 17 15:39:53.951389 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:53.951351    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r972h\" (UniqueName: \"kubernetes.io/projected/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-kube-api-access-r972h\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:53.953719 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:53.953701    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r972h\" (UniqueName: \"kubernetes.io/projected/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-kube-api-access-r972h\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:54.052867 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.052806    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"]
Sep 17 15:39:54.054681 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.054664    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.057226 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.057205    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 15:39:54.057617 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.057596    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:39:54.057680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.057618    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 15:39:54.057680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.057647    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mxjb5\""
Sep 17 15:39:54.057777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.057643    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 15:39:54.066413 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.066393    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"]
Sep 17 15:39:54.153330 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.153308    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57a987c2-12b2-4eeb-89f4-fa5d36855b34-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vtm6n\" (UID: \"57a987c2-12b2-4eeb-89f4-fa5d36855b34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.153463 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.153349    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqgx\" (UniqueName: \"kubernetes.io/projected/57a987c2-12b2-4eeb-89f4-fa5d36855b34-kube-api-access-ppqgx\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vtm6n\" (UID: \"57a987c2-12b2-4eeb-89f4-fa5d36855b34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.153463 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.153371    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57a987c2-12b2-4eeb-89f4-fa5d36855b34-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vtm6n\" (UID: \"57a987c2-12b2-4eeb-89f4-fa5d36855b34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.254253 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.254228    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57a987c2-12b2-4eeb-89f4-fa5d36855b34-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vtm6n\" (UID: \"57a987c2-12b2-4eeb-89f4-fa5d36855b34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.254627 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.254267    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqgx\" (UniqueName: \"kubernetes.io/projected/57a987c2-12b2-4eeb-89f4-fa5d36855b34-kube-api-access-ppqgx\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vtm6n\" (UID: \"57a987c2-12b2-4eeb-89f4-fa5d36855b34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.254627 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.254291    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57a987c2-12b2-4eeb-89f4-fa5d36855b34-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vtm6n\" (UID: \"57a987c2-12b2-4eeb-89f4-fa5d36855b34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.254842 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.254810    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57a987c2-12b2-4eeb-89f4-fa5d36855b34-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vtm6n\" (UID: \"57a987c2-12b2-4eeb-89f4-fa5d36855b34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.256496 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.256477    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57a987c2-12b2-4eeb-89f4-fa5d36855b34-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vtm6n\" (UID: \"57a987c2-12b2-4eeb-89f4-fa5d36855b34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.266471 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.266445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqgx\" (UniqueName: \"kubernetes.io/projected/57a987c2-12b2-4eeb-89f4-fa5d36855b34-kube-api-access-ppqgx\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vtm6n\" (UID: \"57a987c2-12b2-4eeb-89f4-fa5d36855b34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.363028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.362995    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"
Sep 17 15:39:54.630196 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.630055    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n"]
Sep 17 15:39:54.632510 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:39:54.632485    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a987c2_12b2_4eeb_89f4_fa5d36855b34.slice/crio-2e6135ee55cd72c55b2efeeb5f5a24cdce64ac8cb7f7e0cfb4bbc0b215aabce6 WatchSource:0}: Error finding container 2e6135ee55cd72c55b2efeeb5f5a24cdce64ac8cb7f7e0cfb4bbc0b215aabce6: Status 404 returned error can't find the container with id 2e6135ee55cd72c55b2efeeb5f5a24cdce64ac8cb7f7e0cfb4bbc0b215aabce6
Sep 17 15:39:54.857739 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:54.857709    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:54.857883 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:54.857807    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:39:54.857883 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:54.857879    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls podName:1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:56.857864437 +0000 UTC m=+78.748724959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cmkxg" (UID: "1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:39:55.203059 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:55.203029    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n" event={"ID":"57a987c2-12b2-4eeb-89f4-fa5d36855b34","Type":"ContainerStarted","Data":"2e6135ee55cd72c55b2efeeb5f5a24cdce64ac8cb7f7e0cfb4bbc0b215aabce6"}
Sep 17 15:39:55.864159 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:55.864123    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:39:55.864518 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:55.864253    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:39:55.864518 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:55.864315    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls podName:c4eef5bb-eeae-4a79-8362-1b37ca2499f3 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:59.864300276 +0000 UTC m=+81.755160798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-n47fv" (UID: "c4eef5bb-eeae-4a79-8362-1b37ca2499f3") : secret "samples-operator-tls" not found
Sep 17 15:39:56.866415 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:56.866383    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s"]
Sep 17 15:39:56.869312 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:56.869297    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s"
Sep 17 15:39:56.870625 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:56.870602    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2zfs\" (UniqueName: \"kubernetes.io/projected/d543ccd7-be08-42cb-9234-b3f3cf3f84b1-kube-api-access-t2zfs\") pod \"network-check-source-6f8b768455-4wj5s\" (UID: \"d543ccd7-be08-42cb-9234-b3f3cf3f84b1\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s"
Sep 17 15:39:56.870734 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:56.870720    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:39:56.870905 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:56.870887    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:39:56.870985 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:56.870955    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls podName:1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:00.870936132 +0000 UTC m=+82.761796671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cmkxg" (UID: "1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:39:56.872163 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:56.872137    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rtb9p\""
Sep 17 15:39:56.878570 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:56.878539    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s"]
Sep 17 15:39:56.971429 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:56.971367    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2zfs\" (UniqueName: \"kubernetes.io/projected/d543ccd7-be08-42cb-9234-b3f3cf3f84b1-kube-api-access-t2zfs\") pod \"network-check-source-6f8b768455-4wj5s\" (UID: \"d543ccd7-be08-42cb-9234-b3f3cf3f84b1\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s"
Sep 17 15:39:56.979343 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:56.979321    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2zfs\" (UniqueName: \"kubernetes.io/projected/d543ccd7-be08-42cb-9234-b3f3cf3f84b1-kube-api-access-t2zfs\") pod \"network-check-source-6f8b768455-4wj5s\" (UID: \"d543ccd7-be08-42cb-9234-b3f3cf3f84b1\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s"
Sep 17 15:39:57.182078 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:57.182046    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s"
Sep 17 15:39:57.221541 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:57.221458    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n" event={"ID":"57a987c2-12b2-4eeb-89f4-fa5d36855b34","Type":"ContainerStarted","Data":"9fc223582a5b9cf9efc20438e63145c341822a8ccd1566e6bbb07e8e48894131"}
Sep 17 15:39:57.469627 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:57.469574    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n" podStartSLOduration=1.465977923 podStartE2EDuration="3.469554876s" totalImagesPullingTime="2.003576953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:57.240330208 +0000 UTC m=+79.131190749" watchObservedRunningTime="2026-09-17 15:39:57.469554876 +0000 UTC m=+79.360415423"
Sep 17 15:39:57.470333 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:57.470315    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s"]
Sep 17 15:39:57.472542 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:39:57.472492    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd543ccd7_be08_42cb_9234_b3f3cf3f84b1.slice/crio-91688c49b3c91df68f42e9b5643167257bd7ef80737134aa0f4628432e7fcaf5 WatchSource:0}: Error finding container 91688c49b3c91df68f42e9b5643167257bd7ef80737134aa0f4628432e7fcaf5: Status 404 returned error can't find the container with id 91688c49b3c91df68f42e9b5643167257bd7ef80737134aa0f4628432e7fcaf5
Sep 17 15:39:58.130072 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.130040    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s6n66"]
Sep 17 15:39:58.133231 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.133205    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.136139 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.136116    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:39:58.136257 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.136115    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:39:58.136574 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.136559    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:39:58.137193 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.137166    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8w8t9\""
Sep 17 15:39:58.137312 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.137294    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:39:58.145339 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.145319    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s6n66"]
Sep 17 15:39:58.179428 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.179406    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.179538 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.179455    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18308130-5434-45b2-9167-32b5f771ea52-data-volume\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.179538 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.179475    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96dd\" (UniqueName: \"kubernetes.io/projected/18308130-5434-45b2-9167-32b5f771ea52-kube-api-access-f96dd\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.179631 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.179565    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.179631 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.179606    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18308130-5434-45b2-9167-32b5f771ea52-crio-socket\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.226921 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.226888    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s" event={"ID":"d543ccd7-be08-42cb-9234-b3f3cf3f84b1","Type":"ContainerStarted","Data":"8dda6d24b202d66ad56f9785b10b598f74b750d63bbd8d245ac781ab8df2a2b5"}
Sep 17 15:39:58.227030 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.226946    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s" event={"ID":"d543ccd7-be08-42cb-9234-b3f3cf3f84b1","Type":"ContainerStarted","Data":"91688c49b3c91df68f42e9b5643167257bd7ef80737134aa0f4628432e7fcaf5"}
Sep 17 15:39:58.245077 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.245034    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-4wj5s" podStartSLOduration=2.245017477 podStartE2EDuration="2.245017477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:58.244267683 +0000 UTC m=+80.135128222" watchObservedRunningTime="2026-09-17 15:39:58.245017477 +0000 UTC m=+80.135878022"
Sep 17 15:39:58.280363 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.280339    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18308130-5434-45b2-9167-32b5f771ea52-crio-socket\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.280475 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.280394    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.280539 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.280475    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18308130-5434-45b2-9167-32b5f771ea52-crio-socket\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.280539 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:58.280478    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:39:58.280656 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.280550    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18308130-5434-45b2-9167-32b5f771ea52-data-volume\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.280656 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.280593    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96dd\" (UniqueName: \"kubernetes.io/projected/18308130-5434-45b2-9167-32b5f771ea52-kube-api-access-f96dd\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.280755 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.280731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.280861 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:58.280842    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:58.280968 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:58.280845    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:58.780792551 +0000 UTC m=+80.671653106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : configmap "kube-rbac-proxy" not found
Sep 17 15:39:58.280968 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:58.280908    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:58.780899198 +0000 UTC m=+80.671759721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:58.280968 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.280904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18308130-5434-45b2-9167-32b5f771ea52-data-volume\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.289443 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.289414    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96dd\" (UniqueName: \"kubernetes.io/projected/18308130-5434-45b2-9167-32b5f771ea52-kube-api-access-f96dd\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.784160 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.784128    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.784349 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:58.784172    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:58.784349 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:58.784270    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:39:58.784349 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:58.784272    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:58.784349 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:58.784314    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:59.784301284 +0000 UTC m=+81.675161819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : configmap "kube-rbac-proxy" not found
Sep 17 15:39:58.784349 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:58.784326    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:59.784320336 +0000 UTC m=+81.675180857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:59.791991 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:59.791947    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:59.792359 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:59.792011    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:39:59.792359 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:59.792081    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:39:59.792359 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:59.792146    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:01.792131531 +0000 UTC m=+83.682992069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : configmap "kube-rbac-proxy" not found
Sep 17 15:39:59.792359 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:59.792086    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:59.792359 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:59.792220    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:01.792207503 +0000 UTC m=+83.683068025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:59.892582 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:39:59.892549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:39:59.892723 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:59.892694    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:39:59.892759 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:39:59.892752    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls podName:c4eef5bb-eeae-4a79-8362-1b37ca2499f3 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:07.89273824 +0000 UTC m=+89.783598762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-n47fv" (UID: "c4eef5bb-eeae-4a79-8362-1b37ca2499f3") : secret "samples-operator-tls" not found
Sep 17 15:40:00.082377 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:00.082350    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rjjjt_6d538ca1-32a3-4bd3-95f4-44177e3d30ab/dns-node-resolver/0.log"
Sep 17 15:40:00.899967 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:00.899933    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:40:00.900325 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:00.900042    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:40:00.900325 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:00.900095    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls podName:1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:08.900078229 +0000 UTC m=+90.790938754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cmkxg" (UID: "1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:40:01.079125 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:01.079097    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4k758_33b10e70-61bd-4101-9a26-e0349b249102/node-ca/0.log"
Sep 17 15:40:01.806695 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:01.806665    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:40:01.806874 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:01.806751    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:40:01.806874 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:01.806789    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:01.806874 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:01.806871    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:01.806991 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:01.806875    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:05.806858872 +0000 UTC m=+87.697719407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:01.806991 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:01.806923    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:05.806908884 +0000 UTC m=+87.697769406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:01.880335 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:01.880306    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h5tw5_9addd732-ac2d-4f71-9057-def9b3c7fbe5/serve-healthcheck-canary/0.log"
Sep 17 15:40:02.001264 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.001237    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"]
Sep 17 15:40:02.005590 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.005569    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.008205 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.008185    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 15:40:02.008327 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.008275    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 15:40:02.008327 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.008291    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 15:40:02.008438 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.008321    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 15:40:02.009555 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.009536    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 15:40:02.009555 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.009551    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 15:40:02.009703 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.009536    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 15:40:02.017168 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.017146    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"]
Sep 17 15:40:02.108595 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.108575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-ca\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.108686 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.108600    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.108686 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.108621    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.108686 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.108641    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-hub\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.108798 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.108739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2pm8\" (UniqueName: \"kubernetes.io/projected/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-kube-api-access-j2pm8\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.108798 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.108784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.209804 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.209770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2pm8\" (UniqueName: \"kubernetes.io/projected/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-kube-api-access-j2pm8\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.209885 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.209869    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.209943 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.209929    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-ca\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.209985 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.209955    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.210028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.209988    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.210028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.210020    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-hub\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.210759 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.210734    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.212385 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.212360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.212474 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.212395    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.212474 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.212407    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-ca\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.212546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.212521    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-hub\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.219527 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.219504    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2pm8\" (UniqueName: \"kubernetes.io/projected/f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3-kube-api-access-j2pm8\") pod \"cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8\" (UID: \"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.324407 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.324381    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:02.593246 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:02.593193    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"]
Sep 17 15:40:02.595596 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:40:02.595571    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3abd51c_a8ef_4223_9ecb_f8e60d45ceb3.slice/crio-f8065624ab4c3b88afa6f569e3cba52a8644ab3020c49bbf8d0b2a81ba81abc7 WatchSource:0}: Error finding container f8065624ab4c3b88afa6f569e3cba52a8644ab3020c49bbf8d0b2a81ba81abc7: Status 404 returned error can't find the container with id f8065624ab4c3b88afa6f569e3cba52a8644ab3020c49bbf8d0b2a81ba81abc7
Sep 17 15:40:03.247078 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:03.247040    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" event={"ID":"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3","Type":"ContainerStarted","Data":"f8065624ab4c3b88afa6f569e3cba52a8644ab3020c49bbf8d0b2a81ba81abc7"}
Sep 17 15:40:05.842080 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:05.842047    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:40:05.842528 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:05.842115    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:40:05.842528 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:05.842221    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:05.842528 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:05.842298    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.842278319 +0000 UTC m=+95.733138859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:05.842528 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:05.842221    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:05.842528 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:05.842395    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.84237592 +0000 UTC m=+95.733236445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:07.263851 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:07.263805    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" event={"ID":"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3","Type":"ContainerStarted","Data":"4245b537b2fc709a45eeb9801ce5059d96b9bbf5ba59bc5505ddb5135e70b852"}
Sep 17 15:40:07.264207 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:07.263867    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" event={"ID":"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3","Type":"ContainerStarted","Data":"a5ce55e92ca805cf226662fe2a576d408d444dae51f693736d1d15838bcca307"}
Sep 17 15:40:07.959095 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:07.959064    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:40:07.961382 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:07.961360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4eef5bb-eeae-4a79-8362-1b37ca2499f3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n47fv\" (UID: \"c4eef5bb-eeae-4a79-8362-1b37ca2499f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:40:08.005217 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:08.005188    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"
Sep 17 15:40:08.269176 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:08.269109    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" event={"ID":"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3","Type":"ContainerStarted","Data":"fc45f76863d21f146ee519c7c1506e3db2ba623c18cbea94f1f5478069fbbb4b"}
Sep 17 15:40:08.273723 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:08.273698    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv"]
Sep 17 15:40:08.290281 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:08.290238    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" podStartSLOduration=2.745313577 podStartE2EDuration="7.290225036s" totalImagesPullingTime="4.544911459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:08.289819307 +0000 UTC m=+90.180679854" watchObservedRunningTime="2026-09-17 15:40:08.290225036 +0000 UTC m=+90.181085573"
Sep 17 15:40:08.968017 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:08.967979    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:40:08.968192 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:08.968103    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:40:08.968192 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:08.968157    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls podName:1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:24.968143604 +0000 UTC m=+106.859004126 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cmkxg" (UID: "1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:40:09.273716 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:09.273632    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv" event={"ID":"c4eef5bb-eeae-4a79-8362-1b37ca2499f3","Type":"ContainerStarted","Data":"9c6c553d8af5952731bc0ba8e18bb91e58e954f2ae9074e9e5034cb515836041"}
Sep 17 15:40:10.279459 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:10.279429    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv" event={"ID":"c4eef5bb-eeae-4a79-8362-1b37ca2499f3","Type":"ContainerStarted","Data":"744edaa7e920139c3f18c631699e9e0aee04da660236ba6a4e97fbad55f21741"}
Sep 17 15:40:10.279814 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:10.279487    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv" event={"ID":"c4eef5bb-eeae-4a79-8362-1b37ca2499f3","Type":"ContainerStarted","Data":"cae35f861def4a1639387f6afc39d9e658c4252f67d74ac2288d82ee00476d12"}
Sep 17 15:40:10.297407 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:10.297361    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n47fv" podStartSLOduration=16.469888398 podStartE2EDuration="18.297346451s" totalImagesPullingTime="1.827458053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:10.295877154 +0000 UTC m=+92.186737698" watchObservedRunningTime="2026-09-17 15:40:10.297346451 +0000 UTC m=+92.188207010"
Sep 17 15:40:12.327079 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:12.327023    2555 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" podUID="f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 15:40:13.902022 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:13.901985    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:40:13.902392 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:13.902047    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:40:13.902392 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:13.902139    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:13.902392 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:13.902216    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls podName:18308130-5434-45b2-9167-32b5f771ea52 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:29.90219542 +0000 UTC m=+111.793055965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s6n66" (UID: "18308130-5434-45b2-9167-32b5f771ea52") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:13.902673 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:13.902655    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18308130-5434-45b2-9167-32b5f771ea52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:40:15.917370 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:15.917337    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:40:15.917370 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:15.917369    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:40:15.919580 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:15.919556    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/914dd600-d7ac-49f2-b490-5a4d82d748ae-metrics-tls\") pod \"dns-default-p2dt6\" (UID: \"914dd600-d7ac-49f2-b490-5a4d82d748ae\") " pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:40:15.919674 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:15.919610    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"image-registry-64b78c745c-47ztw\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") " pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:40:15.926112 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:15.926093    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9xnvp\""
Sep 17 15:40:15.933560 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:15.933547    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:40:16.200838 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:16.200802    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p2dt6"]
Sep 17 15:40:16.203155 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:40:16.203132    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914dd600_d7ac_49f2_b490_5a4d82d748ae.slice/crio-ba7f7eb1e760b86b8f081c96bf28bf6e8d74c03d5b579112cab7076b0ce69718 WatchSource:0}: Error finding container ba7f7eb1e760b86b8f081c96bf28bf6e8d74c03d5b579112cab7076b0ce69718: Status 404 returned error can't find the container with id ba7f7eb1e760b86b8f081c96bf28bf6e8d74c03d5b579112cab7076b0ce69718
Sep 17 15:40:16.213990 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:16.213970    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w2hrw\""
Sep 17 15:40:16.222430 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:16.222413    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:40:16.298996 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:16.298968    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2dt6" event={"ID":"914dd600-d7ac-49f2-b490-5a4d82d748ae","Type":"ContainerStarted","Data":"ba7f7eb1e760b86b8f081c96bf28bf6e8d74c03d5b579112cab7076b0ce69718"}
Sep 17 15:40:16.492525 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:16.492498    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64b78c745c-47ztw"]
Sep 17 15:40:16.494235 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:40:16.494212    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3f3fb6_88e6_4198_b631_05b2a0f548cc.slice/crio-700289e38fb80457401d1f6d75228278536dbaadab1f76f5d8205cd1704101a8 WatchSource:0}: Error finding container 700289e38fb80457401d1f6d75228278536dbaadab1f76f5d8205cd1704101a8: Status 404 returned error can't find the container with id 700289e38fb80457401d1f6d75228278536dbaadab1f76f5d8205cd1704101a8
Sep 17 15:40:17.304694 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:17.304650    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b78c745c-47ztw" event={"ID":"da3f3fb6-88e6-4198-b631-05b2a0f548cc","Type":"ContainerStarted","Data":"4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473"}
Sep 17 15:40:17.305121 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:17.304714    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b78c745c-47ztw" event={"ID":"da3f3fb6-88e6-4198-b631-05b2a0f548cc","Type":"ContainerStarted","Data":"700289e38fb80457401d1f6d75228278536dbaadab1f76f5d8205cd1704101a8"}
Sep 17 15:40:17.305121 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:17.304737    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:40:17.332473 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:17.332420    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64b78c745c-47ztw" podStartSLOduration=98.332399493 podStartE2EDuration="1m38.332399493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:17.330816655 +0000 UTC m=+99.221677198" watchObservedRunningTime="2026-09-17 15:40:17.332399493 +0000 UTC m=+99.223260039"
Sep 17 15:40:18.309870 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:18.309776    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2dt6" event={"ID":"914dd600-d7ac-49f2-b490-5a4d82d748ae","Type":"ContainerStarted","Data":"132443eb41134a91071aba583866e9d3b19d877f4068a51b8b893fbf8cfe801c"}
Sep 17 15:40:18.310203 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:18.309877    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2dt6" event={"ID":"914dd600-d7ac-49f2-b490-5a4d82d748ae","Type":"ContainerStarted","Data":"8f532750219d83ea03a73059691f3b817163589b236d5f6086d8bfeeab4876a0"}
Sep 17 15:40:18.328547 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:18.328505    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p2dt6" podStartSLOduration=65.610627033 podStartE2EDuration="1m7.328493677s" totalImagesPullingTime="1.717866644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:18.328250067 +0000 UTC m=+100.219110612" watchObservedRunningTime="2026-09-17 15:40:18.328493677 +0000 UTC m=+100.219354221"
Sep 17 15:40:19.312905 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:19.312873    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:40:20.192666 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:20.192637    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6nj4l"
Sep 17 15:40:22.325440 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:22.325399    2555 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" podUID="f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 15:40:24.985078 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:24.985029    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:40:24.987336 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:24.987316    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cmkxg\" (UID: \"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:40:25.218801 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:25.218758    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"
Sep 17 15:40:25.488327 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:25.488296    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg"]
Sep 17 15:40:25.492131 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:40:25.492096    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca0910f_5836_46ac_9e2e_20fbb8d3f7a2.slice/crio-cce0e4dd791c8be4ef6ba62be4d7d75be5ed2d7cfd0df54962c04c6622680147 WatchSource:0}: Error finding container cce0e4dd791c8be4ef6ba62be4d7d75be5ed2d7cfd0df54962c04c6622680147: Status 404 returned error can't find the container with id cce0e4dd791c8be4ef6ba62be4d7d75be5ed2d7cfd0df54962c04c6622680147
Sep 17 15:40:26.335493 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:26.335458    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg" event={"ID":"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2","Type":"ContainerStarted","Data":"cce0e4dd791c8be4ef6ba62be4d7d75be5ed2d7cfd0df54962c04c6622680147"}
Sep 17 15:40:27.345328 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:27.345288    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg" event={"ID":"1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2","Type":"ContainerStarted","Data":"eac33d2206661053903d030ba996d55de04a06a9977a641fc967cf506eb406c2"}
Sep 17 15:40:27.365709 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:27.365659    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cmkxg" podStartSLOduration=33.665143612 podStartE2EDuration="35.365645792s" totalImagesPullingTime="1.70050218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:27.364278343 +0000 UTC m=+109.255138888" watchObservedRunningTime="2026-09-17 15:40:27.365645792 +0000 UTC m=+109.256506335"
Sep 17 15:40:29.317907 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:29.317875    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p2dt6"
Sep 17 15:40:29.918190 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:29.918154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:40:29.920379 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:29.920357    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18308130-5434-45b2-9167-32b5f771ea52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s6n66\" (UID: \"18308130-5434-45b2-9167-32b5f771ea52\") " pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:40:29.942495 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:29.942472    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s6n66"
Sep 17 15:40:30.210797 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:30.210769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s6n66"]
Sep 17 15:40:30.213114 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:40:30.213084    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18308130_5434_45b2_9167_32b5f771ea52.slice/crio-35a45b41d74e5eab2859fc9855f95e12f6856482c1b1a87c7862257abe7619a4 WatchSource:0}: Error finding container 35a45b41d74e5eab2859fc9855f95e12f6856482c1b1a87c7862257abe7619a4: Status 404 returned error can't find the container with id 35a45b41d74e5eab2859fc9855f95e12f6856482c1b1a87c7862257abe7619a4
Sep 17 15:40:30.355483 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:30.355455    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s6n66" event={"ID":"18308130-5434-45b2-9167-32b5f771ea52","Type":"ContainerStarted","Data":"4e175168bd7ba340b0d0efbce70fd93999056ae1f7c4f8bae7c0ef86b0298dae"}
Sep 17 15:40:30.355774 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:30.355506    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s6n66" event={"ID":"18308130-5434-45b2-9167-32b5f771ea52","Type":"ContainerStarted","Data":"35a45b41d74e5eab2859fc9855f95e12f6856482c1b1a87c7862257abe7619a4"}
Sep 17 15:40:31.361481 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:31.361444    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s6n66" event={"ID":"18308130-5434-45b2-9167-32b5f771ea52","Type":"ContainerStarted","Data":"d41b2e2c8c45bf856597b981893d1651467d89404f6fe59dbc0b489caa712401"}
Sep 17 15:40:32.325504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:32.325474    2555 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" podUID="f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 15:40:32.325632 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:32.325537    2555 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8"
Sep 17 15:40:32.326013 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:32.325976    2555 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"fc45f76863d21f146ee519c7c1506e3db2ba623c18cbea94f1f5478069fbbb4b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 15:40:32.326059 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:32.326032    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" podUID="f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3" containerName="service-proxy" containerID="cri-o://fc45f76863d21f146ee519c7c1506e3db2ba623c18cbea94f1f5478069fbbb4b" gracePeriod=30
Sep 17 15:40:32.366196 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:32.366176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s6n66" event={"ID":"18308130-5434-45b2-9167-32b5f771ea52","Type":"ContainerStarted","Data":"f5d502660d487df2df0e3904f2374fe26f961d13c5be5ba7af604ed0fa427cca"}
Sep 17 15:40:32.387697 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:32.387660    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s6n66" podStartSLOduration=32.675847192 podStartE2EDuration="34.387646474s" totalImagesPullingTime="1.711799282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:32.38657092 +0000 UTC m=+114.277431461" watchObservedRunningTime="2026-09-17 15:40:32.387646474 +0000 UTC m=+114.278507018"
Sep 17 15:40:33.372120 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:33.372080    2555 generic.go:356] "Generic (PLEG): container finished" podID="f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3" containerID="fc45f76863d21f146ee519c7c1506e3db2ba623c18cbea94f1f5478069fbbb4b" exitCode=2
Sep 17 15:40:33.372582 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:33.372153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" event={"ID":"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3","Type":"ContainerDied","Data":"fc45f76863d21f146ee519c7c1506e3db2ba623c18cbea94f1f5478069fbbb4b"}
Sep 17 15:40:33.372582 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:33.372195    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d9c9fbb98-8n7g8" event={"ID":"f3abd51c-a8ef-4223-9ecb-f8e60d45ceb3","Type":"ContainerStarted","Data":"7bf8e70832602c3b38de2777fdf51e9ba66868ac631738502353b710a80c09f4"}
Sep 17 15:40:36.228126 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:36.228094    2555 patch_prober.go:28] interesting pod/image-registry-64b78c745c-47ztw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 15:40:36.228530 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:36.228147    2555 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64b78c745c-47ztw" podUID="da3f3fb6-88e6-4198-b631-05b2a0f548cc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:40:38.314487 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:38.314457    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:40:40.155091 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.155059    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sqg8m"]
Sep 17 15:40:40.198751 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.198722    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.201494 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.201470    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:40:40.201632 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.201473    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m94zj\""
Sep 17 15:40:40.201632 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.201479    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:40:40.202530 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.202509    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:40:40.202530 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.202522    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:40:40.287310 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.287284    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b38eede-593c-46ba-9502-ee099ba94a1c-metrics-client-ca\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.287417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.287315    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b38eede-593c-46ba-9502-ee099ba94a1c-root\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.287417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.287340    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b38eede-593c-46ba-9502-ee099ba94a1c-sys\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.287417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.287359    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-textfile\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.287417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.287403    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-tls\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.287582 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.287448    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-wtmp\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.287582 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.287470    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjfl2\" (UniqueName: \"kubernetes.io/projected/0b38eede-593c-46ba-9502-ee099ba94a1c-kube-api-access-cjfl2\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.287582 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.287497    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.287582 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.287522    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388193 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388168    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b38eede-593c-46ba-9502-ee099ba94a1c-metrics-client-ca\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388193 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388195    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b38eede-593c-46ba-9502-ee099ba94a1c-root\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388361 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b38eede-593c-46ba-9502-ee099ba94a1c-sys\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388361 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388227    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-textfile\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388361 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388245    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-tls\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388361 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388291    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b38eede-593c-46ba-9502-ee099ba94a1c-root\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388361 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b38eede-593c-46ba-9502-ee099ba94a1c-sys\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388361 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388337    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-wtmp\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388361 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:40.388344    2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 15:40:40.388680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjfl2\" (UniqueName: \"kubernetes.io/projected/0b38eede-593c-46ba-9502-ee099ba94a1c-kube-api-access-cjfl2\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388680 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:40.388407    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-tls podName:0b38eede-593c-46ba-9502-ee099ba94a1c nodeName:}" failed. No retries permitted until 2026-09-17 15:40:40.888388066 +0000 UTC m=+122.779248602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-tls") pod "node-exporter-sqg8m" (UID: "0b38eede-593c-46ba-9502-ee099ba94a1c") : secret "node-exporter-tls" not found
Sep 17 15:40:40.388680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388434    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-wtmp\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388442    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388472    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388566    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-textfile\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388896 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388866    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b38eede-593c-46ba-9502-ee099ba94a1c-metrics-client-ca\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.388985 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.388970    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.390517 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.390501    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.397966 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.397947    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjfl2\" (UniqueName: \"kubernetes.io/projected/0b38eede-593c-46ba-9502-ee099ba94a1c-kube-api-access-cjfl2\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.891899 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.891865    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-tls\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:40.894139 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:40.894116    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b38eede-593c-46ba-9502-ee099ba94a1c-node-exporter-tls\") pod \"node-exporter-sqg8m\" (UID: \"0b38eede-593c-46ba-9502-ee099ba94a1c\") " pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:41.107836 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:41.107794    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sqg8m"
Sep 17 15:40:41.140722 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:40:41.140698    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b38eede_593c_46ba_9502_ee099ba94a1c.slice/crio-a129e0011fc29c0dc9ed726cb0ccac83ce77b2b55755fe35c277757ddf10d5de WatchSource:0}: Error finding container a129e0011fc29c0dc9ed726cb0ccac83ce77b2b55755fe35c277757ddf10d5de: Status 404 returned error can't find the container with id a129e0011fc29c0dc9ed726cb0ccac83ce77b2b55755fe35c277757ddf10d5de
Sep 17 15:40:41.399091 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:41.399007    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqg8m" event={"ID":"0b38eede-593c-46ba-9502-ee099ba94a1c","Type":"ContainerStarted","Data":"a129e0011fc29c0dc9ed726cb0ccac83ce77b2b55755fe35c277757ddf10d5de"}
Sep 17 15:40:43.406181 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:43.406142    2555 generic.go:356] "Generic (PLEG): container finished" podID="0b38eede-593c-46ba-9502-ee099ba94a1c" containerID="fd2efdb8d16e4b006858cd0ee3cfbeba2dea5cdbc094d3c3bc0bb2b75725785a" exitCode=0
Sep 17 15:40:43.406567 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:43.406190    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqg8m" event={"ID":"0b38eede-593c-46ba-9502-ee099ba94a1c","Type":"ContainerDied","Data":"fd2efdb8d16e4b006858cd0ee3cfbeba2dea5cdbc094d3c3bc0bb2b75725785a"}
Sep 17 15:40:44.412182 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:44.412148    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqg8m" event={"ID":"0b38eede-593c-46ba-9502-ee099ba94a1c","Type":"ContainerStarted","Data":"315bc69aa47e50cc4b6b7ae8404b7ac9e37c5840caeca1f285afa0dc3208035e"}
Sep 17 15:40:44.412584 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:44.412199    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqg8m" event={"ID":"0b38eede-593c-46ba-9502-ee099ba94a1c","Type":"ContainerStarted","Data":"cafaea3d961962eca8fa43a9eacdd08c0d683d0cd63ca7f443604eb31829f793"}
Sep 17 15:40:44.440516 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:44.440469    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sqg8m" podStartSLOduration=3.056684419 podStartE2EDuration="4.440453934s" totalImagesPullingTime="1.323845576s" totalInitContainerRuntime="59.923939ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:44.440267792 +0000 UTC m=+126.331128392" watchObservedRunningTime="2026-09-17 15:40:44.440453934 +0000 UTC m=+126.331314478"
Sep 17 15:40:44.920736 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:44.920706    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"]
Sep 17 15:40:44.922898 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:44.922881    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:44.925347 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:44.925324    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 15:40:44.925477 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:44.925457    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j9mlh\""
Sep 17 15:40:44.932362 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:44.932339    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"]
Sep 17 15:40:45.023233 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:45.023209    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q28j\" (UniqueName: \"kubernetes.io/projected/e4be7e13-5d73-4f12-8333-e4ae9dcac435-kube-api-access-5q28j\") pod \"monitoring-plugin-5d85778b9d-gchxf\" (UID: \"e4be7e13-5d73-4f12-8333-e4ae9dcac435\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:45.023333 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:45.023237    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4be7e13-5d73-4f12-8333-e4ae9dcac435-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gchxf\" (UID: \"e4be7e13-5d73-4f12-8333-e4ae9dcac435\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:45.123607 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:45.123587    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4be7e13-5d73-4f12-8333-e4ae9dcac435-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gchxf\" (UID: \"e4be7e13-5d73-4f12-8333-e4ae9dcac435\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:45.123695 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:45.123657    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q28j\" (UniqueName: \"kubernetes.io/projected/e4be7e13-5d73-4f12-8333-e4ae9dcac435-kube-api-access-5q28j\") pod \"monitoring-plugin-5d85778b9d-gchxf\" (UID: \"e4be7e13-5d73-4f12-8333-e4ae9dcac435\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:45.123734 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:45.123716    2555 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 15:40:45.123780 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:40:45.123770    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4be7e13-5d73-4f12-8333-e4ae9dcac435-monitoring-plugin-cert podName:e4be7e13-5d73-4f12-8333-e4ae9dcac435 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:45.62375531 +0000 UTC m=+127.514615852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/e4be7e13-5d73-4f12-8333-e4ae9dcac435-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-gchxf" (UID: "e4be7e13-5d73-4f12-8333-e4ae9dcac435") : secret "monitoring-plugin-cert" not found
Sep 17 15:40:45.134339 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:45.134313    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q28j\" (UniqueName: \"kubernetes.io/projected/e4be7e13-5d73-4f12-8333-e4ae9dcac435-kube-api-access-5q28j\") pod \"monitoring-plugin-5d85778b9d-gchxf\" (UID: \"e4be7e13-5d73-4f12-8333-e4ae9dcac435\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:45.627445 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:45.627412    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4be7e13-5d73-4f12-8333-e4ae9dcac435-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gchxf\" (UID: \"e4be7e13-5d73-4f12-8333-e4ae9dcac435\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:45.629810 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:45.629786    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4be7e13-5d73-4f12-8333-e4ae9dcac435-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gchxf\" (UID: \"e4be7e13-5d73-4f12-8333-e4ae9dcac435\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:45.831630 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:45.831609    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:46.097741 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:46.097721    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"]
Sep 17 15:40:46.099923 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:40:46.099894    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4be7e13_5d73_4f12_8333_e4ae9dcac435.slice/crio-7adc6843b4c30fbd9f54ae2cf4f900b1a357a54f8aca2ea75fb302d583a6415a WatchSource:0}: Error finding container 7adc6843b4c30fbd9f54ae2cf4f900b1a357a54f8aca2ea75fb302d583a6415a: Status 404 returned error can't find the container with id 7adc6843b4c30fbd9f54ae2cf4f900b1a357a54f8aca2ea75fb302d583a6415a
Sep 17 15:40:46.418898 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:46.418810    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf" event={"ID":"e4be7e13-5d73-4f12-8333-e4ae9dcac435","Type":"ContainerStarted","Data":"7adc6843b4c30fbd9f54ae2cf4f900b1a357a54f8aca2ea75fb302d583a6415a"}
Sep 17 15:40:47.424023 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:47.423994    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf" event={"ID":"e4be7e13-5d73-4f12-8333-e4ae9dcac435","Type":"ContainerStarted","Data":"a99764924a3741d151c87aeb45ac4ff2adcbb4a5edba310dbb2edc4f7d2b8d4a"}
Sep 17 15:40:47.424327 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:47.424179    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:47.428796 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:47.428776    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf"
Sep 17 15:40:47.441894 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:47.441873    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:40:47.443928 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:47.443906    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf8474f-71ab-45db-bfaa-e4953f520025-metrics-certs\") pod \"network-metrics-daemon-b2879\" (UID: \"2cf8474f-71ab-45db-bfaa-e4953f520025\") " pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:40:47.444442 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:47.444397    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gchxf" podStartSLOduration=2.243471249 podStartE2EDuration="3.444377959s" totalImagesPullingTime="1.20090671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:47.443120875 +0000 UTC m=+129.333981430" watchObservedRunningTime="2026-09-17 15:40:47.444377959 +0000 UTC m=+129.335238499"
Sep 17 15:40:47.501468 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:47.501442    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j5497\""
Sep 17 15:40:47.508594 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:47.508578    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b2879"
Sep 17 15:40:47.776634 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:47.776604    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b2879"]
Sep 17 15:40:47.779317 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:40:47.779290    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf8474f_71ab_45db_bfaa_e4953f520025.slice/crio-ec58a778f2c8b40a655488e04ce0038c8a45640d4e67fa1300e9cc92270af587 WatchSource:0}: Error finding container ec58a778f2c8b40a655488e04ce0038c8a45640d4e67fa1300e9cc92270af587: Status 404 returned error can't find the container with id ec58a778f2c8b40a655488e04ce0038c8a45640d4e67fa1300e9cc92270af587
Sep 17 15:40:48.429634 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:48.429593    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b2879" event={"ID":"2cf8474f-71ab-45db-bfaa-e4953f520025","Type":"ContainerStarted","Data":"ec58a778f2c8b40a655488e04ce0038c8a45640d4e67fa1300e9cc92270af587"}
Sep 17 15:40:49.435522 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:49.435482    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b2879" event={"ID":"2cf8474f-71ab-45db-bfaa-e4953f520025","Type":"ContainerStarted","Data":"52c08189460376cb5376bf47620b261a5238b55d6f5e22838578c2a6f4625625"}
Sep 17 15:40:49.435979 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:49.435565    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b2879" event={"ID":"2cf8474f-71ab-45db-bfaa-e4953f520025","Type":"ContainerStarted","Data":"304c2965761ceec6b182061e119a82cb906bb4325f60bffa2889d547d161ea55"}
Sep 17 15:40:49.456035 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:49.455992    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b2879" podStartSLOduration=129.673768912 podStartE2EDuration="2m10.455977142s" totalImagesPullingTime="782.20823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:49.45464726 +0000 UTC m=+131.345507803" watchObservedRunningTime="2026-09-17 15:40:49.455977142 +0000 UTC m=+131.346837685"
Sep 17 15:40:54.396125 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.396093    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8444994899-jsdbk"]
Sep 17 15:40:54.399416 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.399395    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.402050 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.402030    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 15:40:54.403204 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.403182    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 15:40:54.403300 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.403184    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 15:40:54.403533 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.403511    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 15:40:54.403642 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.403577    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 15:40:54.403642 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.403521    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 15:40:54.403744 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.403681    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t7h6z\""
Sep 17 15:40:54.403840 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.403812    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 15:40:54.408020 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.407999    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 15:40:54.408795 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.408775    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8444994899-jsdbk"]
Sep 17 15:40:54.489237 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.489200    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-oauth-serving-cert\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.489237 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.489243    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-oauth-config\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.489394 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.489260    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hs9q\" (UniqueName: \"kubernetes.io/projected/381af671-444a-4e93-a322-8d01b9774064-kube-api-access-4hs9q\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.489394 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.489326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-service-ca\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.489394 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.489349    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-console-config\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.489394 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.489383    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-serving-cert\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.489517 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.489413    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-trusted-ca-bundle\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.590606 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.590577    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-oauth-serving-cert\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.590722 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.590616    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-oauth-config\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.590722 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.590640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hs9q\" (UniqueName: \"kubernetes.io/projected/381af671-444a-4e93-a322-8d01b9774064-kube-api-access-4hs9q\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.590722 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.590676    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-service-ca\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.590722 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.590699    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-console-config\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.590965 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.590721    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-serving-cert\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.590965 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.590770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-trusted-ca-bundle\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.591398 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.591375    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-service-ca\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.591398 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.591392    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-oauth-serving-cert\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.591556 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.591403    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-console-config\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.591638 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.591615    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-trusted-ca-bundle\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.593125 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.593097    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-oauth-config\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.593221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.593204    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-serving-cert\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.599567 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.599549    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hs9q\" (UniqueName: \"kubernetes.io/projected/381af671-444a-4e93-a322-8d01b9774064-kube-api-access-4hs9q\") pod \"console-8444994899-jsdbk\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") " pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.709309 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.709255    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:40:54.982118 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:54.982094    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8444994899-jsdbk"]
Sep 17 15:40:54.984273 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:40:54.984247    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381af671_444a_4e93_a322_8d01b9774064.slice/crio-a0796ad409ce8d3173ff5e732c5535ae55086e99809eb4499c284b7c77f26722 WatchSource:0}: Error finding container a0796ad409ce8d3173ff5e732c5535ae55086e99809eb4499c284b7c77f26722: Status 404 returned error can't find the container with id a0796ad409ce8d3173ff5e732c5535ae55086e99809eb4499c284b7c77f26722
Sep 17 15:40:55.183034 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:55.183005    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64b78c745c-47ztw"]
Sep 17 15:40:55.458971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:55.458931    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8444994899-jsdbk" event={"ID":"381af671-444a-4e93-a322-8d01b9774064","Type":"ContainerStarted","Data":"a0796ad409ce8d3173ff5e732c5535ae55086e99809eb4499c284b7c77f26722"}
Sep 17 15:40:58.471986 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:58.471948    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8444994899-jsdbk" event={"ID":"381af671-444a-4e93-a322-8d01b9774064","Type":"ContainerStarted","Data":"fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b"}
Sep 17 15:40:58.491773 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:40:58.491726    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8444994899-jsdbk" podStartSLOduration=1.607699698 podStartE2EDuration="4.491709519s" totalImagesPullingTime="2.884009821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:58.49090357 +0000 UTC m=+140.381764114" watchObservedRunningTime="2026-09-17 15:40:58.491709519 +0000 UTC m=+140.382570063"
Sep 17 15:41:04.709886 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:04.709855    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:41:04.710283 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:04.709901    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:41:04.714580 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:04.714555    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:41:05.498445 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:05.498414    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:41:09.188783 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:09.188748    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h5tw5_9addd732-ac2d-4f71-9057-def9b3c7fbe5/serve-healthcheck-canary/0.log"
Sep 17 15:41:20.202693 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.202640    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64b78c745c-47ztw" podUID="da3f3fb6-88e6-4198-b631-05b2a0f548cc" containerName="registry" containerID="cri-o://4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473" gracePeriod=30
Sep 17 15:41:20.454334 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.453970    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:41:20.555726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.555696    2555 generic.go:356] "Generic (PLEG): container finished" podID="da3f3fb6-88e6-4198-b631-05b2a0f548cc" containerID="4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473" exitCode=0
Sep 17 15:41:20.555948 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.555756    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b78c745c-47ztw" event={"ID":"da3f3fb6-88e6-4198-b631-05b2a0f548cc","Type":"ContainerDied","Data":"4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473"}
Sep 17 15:41:20.555948 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.555793    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b78c745c-47ztw" event={"ID":"da3f3fb6-88e6-4198-b631-05b2a0f548cc","Type":"ContainerDied","Data":"700289e38fb80457401d1f6d75228278536dbaadab1f76f5d8205cd1704101a8"}
Sep 17 15:41:20.555948 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.555813    2555 scope.go:117] "RemoveContainer" containerID="4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473"
Sep 17 15:41:20.555948 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.555758    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b78c745c-47ztw"
Sep 17 15:41:20.564561 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.564540    2555 scope.go:117] "RemoveContainer" containerID="4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473"
Sep 17 15:41:20.564867 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:41:20.564842    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473\": container with ID starting with 4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473 not found: ID does not exist" containerID="4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473"
Sep 17 15:41:20.564938 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.564876    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473"} err="failed to get container status \"4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473\": rpc error: code = NotFound desc = could not find container \"4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473\": container with ID starting with 4202c9d916d5f19421fd0960af6b8cd232fa9b3c361d64eca21cd93b03855473 not found: ID does not exist"
Sep 17 15:41:20.573141 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.573115    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-bound-sa-token\") pod \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") "
Sep 17 15:41:20.573319 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.573292    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tjcm\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-kube-api-access-8tjcm\") pod \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") "
Sep 17 15:41:20.573411 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.573342    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-trusted-ca\") pod \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") "
Sep 17 15:41:20.573411 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.573395    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") pod \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") "
Sep 17 15:41:20.573519 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.573435    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3f3fb6-88e6-4198-b631-05b2a0f548cc-ca-trust-extracted\") pod \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") "
Sep 17 15:41:20.573519 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.573476    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-image-registry-private-configuration\") pod \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") "
Sep 17 15:41:20.573622 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.573528    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-certificates\") pod \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") "
Sep 17 15:41:20.573622 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.573600    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-installation-pull-secrets\") pod \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\" (UID: \"da3f3fb6-88e6-4198-b631-05b2a0f548cc\") "
Sep 17 15:41:20.574038 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.573989    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "da3f3fb6-88e6-4198-b631-05b2a0f548cc" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:41:20.574038 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.574017    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "da3f3fb6-88e6-4198-b631-05b2a0f548cc" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:41:20.575943 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.575889    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-kube-api-access-8tjcm" (OuterVolumeSpecName: "kube-api-access-8tjcm") pod "da3f3fb6-88e6-4198-b631-05b2a0f548cc" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc"). InnerVolumeSpecName "kube-api-access-8tjcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:41:20.576099 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.576076    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "da3f3fb6-88e6-4198-b631-05b2a0f548cc" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:41:20.576179 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.576153    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "da3f3fb6-88e6-4198-b631-05b2a0f548cc" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:41:20.576325 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.576285    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "da3f3fb6-88e6-4198-b631-05b2a0f548cc" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:41:20.577915 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.577880    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "da3f3fb6-88e6-4198-b631-05b2a0f548cc" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:41:20.583760 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.583738    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3f3fb6-88e6-4198-b631-05b2a0f548cc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "da3f3fb6-88e6-4198-b631-05b2a0f548cc" (UID: "da3f3fb6-88e6-4198-b631-05b2a0f548cc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:41:20.674428 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.674386    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-installation-pull-secrets\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:41:20.674428 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.674430    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-bound-sa-token\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:41:20.674730 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.674448    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tjcm\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-kube-api-access-8tjcm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:41:20.674730 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.674464    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-trusted-ca\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:41:20.674730 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.674479    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-tls\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:41:20.674730 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.674492    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3f3fb6-88e6-4198-b631-05b2a0f548cc-ca-trust-extracted\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:41:20.674730 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.674510    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3f3fb6-88e6-4198-b631-05b2a0f548cc-image-registry-private-configuration\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:41:20.674730 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.674526    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3f3fb6-88e6-4198-b631-05b2a0f548cc-registry-certificates\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:41:20.873213 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.873182    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64b78c745c-47ztw"]
Sep 17 15:41:20.877396 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:20.877374    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64b78c745c-47ztw"]
Sep 17 15:41:22.693209 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:22.693175    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3f3fb6-88e6-4198-b631-05b2a0f548cc" path="/var/lib/kubelet/pods/da3f3fb6-88e6-4198-b631-05b2a0f548cc/volumes"
Sep 17 15:41:32.598766 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:32.598728    2555 generic.go:356] "Generic (PLEG): container finished" podID="57a987c2-12b2-4eeb-89f4-fa5d36855b34" containerID="9fc223582a5b9cf9efc20438e63145c341822a8ccd1566e6bbb07e8e48894131" exitCode=0
Sep 17 15:41:32.599186 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:32.598804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n" event={"ID":"57a987c2-12b2-4eeb-89f4-fa5d36855b34","Type":"ContainerDied","Data":"9fc223582a5b9cf9efc20438e63145c341822a8ccd1566e6bbb07e8e48894131"}
Sep 17 15:41:32.599186 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:32.599138    2555 scope.go:117] "RemoveContainer" containerID="9fc223582a5b9cf9efc20438e63145c341822a8ccd1566e6bbb07e8e48894131"
Sep 17 15:41:33.604162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:41:33.604123    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vtm6n" event={"ID":"57a987c2-12b2-4eeb-89f4-fa5d36855b34","Type":"ContainerStarted","Data":"1260d4afe163e0172eef4c71f49a88bf968c80808b6552a8bc1444184b3c3b52"}
Sep 17 15:42:08.186440 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.186405    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64945d586d-b4gxt"]
Sep 17 15:42:08.186902 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.186632    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3f3fb6-88e6-4198-b631-05b2a0f548cc" containerName="registry"
Sep 17 15:42:08.186902 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.186642    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3f3fb6-88e6-4198-b631-05b2a0f548cc" containerName="registry"
Sep 17 15:42:08.186902 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.186690    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3f3fb6-88e6-4198-b631-05b2a0f548cc" containerName="registry"
Sep 17 15:42:08.188447 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.188432    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.200872 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.200848    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64945d586d-b4gxt"]
Sep 17 15:42:08.273937 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.273909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-service-ca\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.273937 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.273934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-trusted-ca-bundle\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.274079 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.273952    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-oauth-config\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.274079 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.273972    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-config\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.274079 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.274049    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbx5k\" (UniqueName: \"kubernetes.io/projected/aa6f84fb-c69f-4c1d-967d-ee53355f545c-kube-api-access-qbx5k\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.274193 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.274086    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-serving-cert\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.274193 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.274108    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-oauth-serving-cert\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.374336 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.374310    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-serving-cert\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.374423 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.374341    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-oauth-serving-cert\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.374423 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.374360    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-service-ca\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.374423 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.374376    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-trusted-ca-bundle\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.374423 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.374392    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-oauth-config\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.374423 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.374413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-config\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.374631 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.374452    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbx5k\" (UniqueName: \"kubernetes.io/projected/aa6f84fb-c69f-4c1d-967d-ee53355f545c-kube-api-access-qbx5k\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.375366 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.375306    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-oauth-serving-cert\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.375587 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.375516    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-trusted-ca-bundle\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.379601 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.376174    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-config\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.379601 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.376178    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-service-ca\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.379601 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.379571    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-oauth-config\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.380742 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.380722    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-serving-cert\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.383513 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.383488    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbx5k\" (UniqueName: \"kubernetes.io/projected/aa6f84fb-c69f-4c1d-967d-ee53355f545c-kube-api-access-qbx5k\") pod \"console-64945d586d-b4gxt\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") " pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.497024 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.496965    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:08.769167 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:08.769142    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64945d586d-b4gxt"]
Sep 17 15:42:08.771570 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:42:08.771546    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6f84fb_c69f_4c1d_967d_ee53355f545c.slice/crio-3dc812ae86a77e4967e010315e53ce915438795f438e69310d5f84fa0994955f WatchSource:0}: Error finding container 3dc812ae86a77e4967e010315e53ce915438795f438e69310d5f84fa0994955f: Status 404 returned error can't find the container with id 3dc812ae86a77e4967e010315e53ce915438795f438e69310d5f84fa0994955f
Sep 17 15:42:09.724010 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:09.723977    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64945d586d-b4gxt" event={"ID":"aa6f84fb-c69f-4c1d-967d-ee53355f545c","Type":"ContainerStarted","Data":"3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342"}
Sep 17 15:42:09.724370 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:09.724021    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64945d586d-b4gxt" event={"ID":"aa6f84fb-c69f-4c1d-967d-ee53355f545c","Type":"ContainerStarted","Data":"3dc812ae86a77e4967e010315e53ce915438795f438e69310d5f84fa0994955f"}
Sep 17 15:42:09.743714 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:09.743636    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64945d586d-b4gxt" podStartSLOduration=1.743620704 podStartE2EDuration="1.743620704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:09.742101186 +0000 UTC m=+211.632961742" watchObservedRunningTime="2026-09-17 15:42:09.743620704 +0000 UTC m=+211.634481248"
Sep 17 15:42:18.497089 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:18.497058    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:18.497458 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:18.497319    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:18.501819 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:18.501800    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:18.756538 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:18.756474    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:42:18.826996 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:18.826962    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8444994899-jsdbk"]
Sep 17 15:42:43.845661 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:43.845607    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8444994899-jsdbk" podUID="381af671-444a-4e93-a322-8d01b9774064" containerName="console" containerID="cri-o://fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b" gracePeriod=15
Sep 17 15:42:44.077901 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.077879    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:42:44.205681 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.205601    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-serving-cert\") pod \"381af671-444a-4e93-a322-8d01b9774064\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") "
Sep 17 15:42:44.205681 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.205645    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-trusted-ca-bundle\") pod \"381af671-444a-4e93-a322-8d01b9774064\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") "
Sep 17 15:42:44.205681 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.205669    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hs9q\" (UniqueName: \"kubernetes.io/projected/381af671-444a-4e93-a322-8d01b9774064-kube-api-access-4hs9q\") pod \"381af671-444a-4e93-a322-8d01b9774064\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") "
Sep 17 15:42:44.205681 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.205688    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-oauth-serving-cert\") pod \"381af671-444a-4e93-a322-8d01b9774064\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") "
Sep 17 15:42:44.206026 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.205734    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-oauth-config\") pod \"381af671-444a-4e93-a322-8d01b9774064\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") "
Sep 17 15:42:44.206026 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.205781    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-console-config\") pod \"381af671-444a-4e93-a322-8d01b9774064\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") "
Sep 17 15:42:44.206026 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.205812    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-service-ca\") pod \"381af671-444a-4e93-a322-8d01b9774064\" (UID: \"381af671-444a-4e93-a322-8d01b9774064\") "
Sep 17 15:42:44.206224 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.206200    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "381af671-444a-4e93-a322-8d01b9774064" (UID: "381af671-444a-4e93-a322-8d01b9774064"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:44.206298 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.206240    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "381af671-444a-4e93-a322-8d01b9774064" (UID: "381af671-444a-4e93-a322-8d01b9774064"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:44.206358 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.206295    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-console-config" (OuterVolumeSpecName: "console-config") pod "381af671-444a-4e93-a322-8d01b9774064" (UID: "381af671-444a-4e93-a322-8d01b9774064"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:44.206358 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.206301    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-service-ca" (OuterVolumeSpecName: "service-ca") pod "381af671-444a-4e93-a322-8d01b9774064" (UID: "381af671-444a-4e93-a322-8d01b9774064"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:44.207960 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.207933    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "381af671-444a-4e93-a322-8d01b9774064" (UID: "381af671-444a-4e93-a322-8d01b9774064"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:44.208056 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.207957    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "381af671-444a-4e93-a322-8d01b9774064" (UID: "381af671-444a-4e93-a322-8d01b9774064"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:44.208056 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.207974    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381af671-444a-4e93-a322-8d01b9774064-kube-api-access-4hs9q" (OuterVolumeSpecName: "kube-api-access-4hs9q") pod "381af671-444a-4e93-a322-8d01b9774064" (UID: "381af671-444a-4e93-a322-8d01b9774064"). InnerVolumeSpecName "kube-api-access-4hs9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:44.307060 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.307023    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-oauth-config\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:42:44.307060 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.307054    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-console-config\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:42:44.307060 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.307064    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-service-ca\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:42:44.307060 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.307072    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/381af671-444a-4e93-a322-8d01b9774064-console-serving-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:42:44.307289 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.307080    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-trusted-ca-bundle\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:42:44.307289 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.307089    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hs9q\" (UniqueName: \"kubernetes.io/projected/381af671-444a-4e93-a322-8d01b9774064-kube-api-access-4hs9q\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:42:44.307289 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.307097    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/381af671-444a-4e93-a322-8d01b9774064-oauth-serving-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:42:44.842667 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.842630    2555 generic.go:356] "Generic (PLEG): container finished" podID="381af671-444a-4e93-a322-8d01b9774064" containerID="fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b" exitCode=0
Sep 17 15:42:44.842815 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.842675    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8444994899-jsdbk"
Sep 17 15:42:44.842815 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.842706    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8444994899-jsdbk" event={"ID":"381af671-444a-4e93-a322-8d01b9774064","Type":"ContainerDied","Data":"fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b"}
Sep 17 15:42:44.842815 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.842747    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8444994899-jsdbk" event={"ID":"381af671-444a-4e93-a322-8d01b9774064","Type":"ContainerDied","Data":"a0796ad409ce8d3173ff5e732c5535ae55086e99809eb4499c284b7c77f26722"}
Sep 17 15:42:44.842815 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.842769    2555 scope.go:117] "RemoveContainer" containerID="fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b"
Sep 17 15:42:44.850541 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.850264    2555 scope.go:117] "RemoveContainer" containerID="fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b"
Sep 17 15:42:44.850752 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:42:44.850554    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b\": container with ID starting with fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b not found: ID does not exist" containerID="fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b"
Sep 17 15:42:44.850752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.850582    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b"} err="failed to get container status \"fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b\": rpc error: code = NotFound desc = could not find container \"fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b\": container with ID starting with fd24821349d1a24260ae9cb8a32bdc3981f42ee46a51b4095a49d1439e49d98b not found: ID does not exist"
Sep 17 15:42:44.860373 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.860344    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8444994899-jsdbk"]
Sep 17 15:42:44.864491 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:44.864471    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8444994899-jsdbk"]
Sep 17 15:42:46.692927 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:42:46.692894    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381af671-444a-4e93-a322-8d01b9774064" path="/var/lib/kubelet/pods/381af671-444a-4e93-a322-8d01b9774064/volumes"
Sep 17 15:43:17.190519 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.190475    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fbc8b4559-m5qpc"]
Sep 17 15:43:17.191094 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.190882    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381af671-444a-4e93-a322-8d01b9774064" containerName="console"
Sep 17 15:43:17.191094 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.190904    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="381af671-444a-4e93-a322-8d01b9774064" containerName="console"
Sep 17 15:43:17.191094 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.190988    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="381af671-444a-4e93-a322-8d01b9774064" containerName="console"
Sep 17 15:43:17.192965 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.192940    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.205723 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.205697    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbc8b4559-m5qpc"]
Sep 17 15:43:17.317964 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.317932    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-service-ca\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.318080 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.317969    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-trusted-ca-bundle\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.318080 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.318005    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-console-config\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.318080 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.318031    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-oauth-config\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.318080 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.318076    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-serving-cert\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.318277 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.318133    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-oauth-serving-cert\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.318277 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.318159    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht4b2\" (UniqueName: \"kubernetes.io/projected/a4977f4a-30f4-42c4-a42b-700523f4da39-kube-api-access-ht4b2\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.418617 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.418591    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-service-ca\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.418703 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.418621    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-trusted-ca-bundle\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.418703 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.418645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-console-config\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.418703 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.418660    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-oauth-config\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.418888 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.418799    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-serving-cert\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.418888 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.418870    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-oauth-serving-cert\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.418999 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.418893    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht4b2\" (UniqueName: \"kubernetes.io/projected/a4977f4a-30f4-42c4-a42b-700523f4da39-kube-api-access-ht4b2\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.419494 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.419464    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-service-ca\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.419494 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.419480    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-console-config\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.419727 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.419701    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-oauth-serving-cert\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.419803 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.419787    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-trusted-ca-bundle\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.421266 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.421249    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-oauth-config\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.421399 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.421381    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-serving-cert\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.427754 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.427731    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht4b2\" (UniqueName: \"kubernetes.io/projected/a4977f4a-30f4-42c4-a42b-700523f4da39-kube-api-access-ht4b2\") pod \"console-7fbc8b4559-m5qpc\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") " pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.502942 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.502892    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:17.775493 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.775414    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbc8b4559-m5qpc"]
Sep 17 15:43:17.960943 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.960903    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbc8b4559-m5qpc" event={"ID":"a4977f4a-30f4-42c4-a42b-700523f4da39","Type":"ContainerStarted","Data":"b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91"}
Sep 17 15:43:17.961103 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.960971    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbc8b4559-m5qpc" event={"ID":"a4977f4a-30f4-42c4-a42b-700523f4da39","Type":"ContainerStarted","Data":"ffa508ce6842e66edda7bd54fd81e24585ab2361db058608a8bc314c2885d998"}
Sep 17 15:43:17.982586 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:17.982540    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fbc8b4559-m5qpc" podStartSLOduration=0.982523279 podStartE2EDuration="982.523279ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:43:17.980961756 +0000 UTC m=+279.871822299" watchObservedRunningTime="2026-09-17 15:43:17.982523279 +0000 UTC m=+279.873383822"
Sep 17 15:43:27.503353 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:27.503321    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:27.503743 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:27.503440    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:27.508110 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:27.508086    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:28.004434 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:28.004407    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:43:28.050053 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:28.050023    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64945d586d-b4gxt"]
Sep 17 15:43:38.596676 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:38.596650    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:43:38.601319 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:38.601296    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:43:38.601559 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:38.601542    2555 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 15:43:53.068967 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.068922    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64945d586d-b4gxt" podUID="aa6f84fb-c69f-4c1d-967d-ee53355f545c" containerName="console" containerID="cri-o://3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342" gracePeriod=15
Sep 17 15:43:53.305008 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.304985    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:43:53.465158 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465120    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-oauth-config\") pod \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") "
Sep 17 15:43:53.465279 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465171    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-oauth-serving-cert\") pod \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") "
Sep 17 15:43:53.465279 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465196    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-serving-cert\") pod \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") "
Sep 17 15:43:53.465367 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465307    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-config\") pod \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") "
Sep 17 15:43:53.465367 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465345    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbx5k\" (UniqueName: \"kubernetes.io/projected/aa6f84fb-c69f-4c1d-967d-ee53355f545c-kube-api-access-qbx5k\") pod \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") "
Sep 17 15:43:53.465468 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465408    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-service-ca\") pod \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") "
Sep 17 15:43:53.465468 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465456    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-trusted-ca-bundle\") pod \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\" (UID: \"aa6f84fb-c69f-4c1d-967d-ee53355f545c\") "
Sep 17 15:43:53.465577 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465552    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aa6f84fb-c69f-4c1d-967d-ee53355f545c" (UID: "aa6f84fb-c69f-4c1d-967d-ee53355f545c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:43:53.465780 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465743    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-oauth-serving-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:43:53.465908 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465780    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-config" (OuterVolumeSpecName: "console-config") pod "aa6f84fb-c69f-4c1d-967d-ee53355f545c" (UID: "aa6f84fb-c69f-4c1d-967d-ee53355f545c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:43:53.465962 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465891    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-service-ca" (OuterVolumeSpecName: "service-ca") pod "aa6f84fb-c69f-4c1d-967d-ee53355f545c" (UID: "aa6f84fb-c69f-4c1d-967d-ee53355f545c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:43:53.465962 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.465916    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aa6f84fb-c69f-4c1d-967d-ee53355f545c" (UID: "aa6f84fb-c69f-4c1d-967d-ee53355f545c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:43:53.467352 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.467326    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aa6f84fb-c69f-4c1d-967d-ee53355f545c" (UID: "aa6f84fb-c69f-4c1d-967d-ee53355f545c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:43:53.467450 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.467420    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6f84fb-c69f-4c1d-967d-ee53355f545c-kube-api-access-qbx5k" (OuterVolumeSpecName: "kube-api-access-qbx5k") pod "aa6f84fb-c69f-4c1d-967d-ee53355f545c" (UID: "aa6f84fb-c69f-4c1d-967d-ee53355f545c"). InnerVolumeSpecName "kube-api-access-qbx5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:43:53.467450 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.467429    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aa6f84fb-c69f-4c1d-967d-ee53355f545c" (UID: "aa6f84fb-c69f-4c1d-967d-ee53355f545c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:43:53.565999 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.565977    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-serving-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:43:53.565999 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.565998    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-config\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:43:53.566114 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.566007    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbx5k\" (UniqueName: \"kubernetes.io/projected/aa6f84fb-c69f-4c1d-967d-ee53355f545c-kube-api-access-qbx5k\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:43:53.566114 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.566016    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-service-ca\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:43:53.566114 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.566024    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6f84fb-c69f-4c1d-967d-ee53355f545c-trusted-ca-bundle\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:43:53.566114 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:53.566032    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa6f84fb-c69f-4c1d-967d-ee53355f545c-console-oauth-config\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:43:54.089391 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:54.089361    2555 generic.go:356] "Generic (PLEG): container finished" podID="aa6f84fb-c69f-4c1d-967d-ee53355f545c" containerID="3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342" exitCode=0
Sep 17 15:43:54.089764 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:54.089406    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64945d586d-b4gxt"
Sep 17 15:43:54.089764 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:54.089433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64945d586d-b4gxt" event={"ID":"aa6f84fb-c69f-4c1d-967d-ee53355f545c","Type":"ContainerDied","Data":"3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342"}
Sep 17 15:43:54.089764 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:54.089471    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64945d586d-b4gxt" event={"ID":"aa6f84fb-c69f-4c1d-967d-ee53355f545c","Type":"ContainerDied","Data":"3dc812ae86a77e4967e010315e53ce915438795f438e69310d5f84fa0994955f"}
Sep 17 15:43:54.089764 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:54.089491    2555 scope.go:117] "RemoveContainer" containerID="3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342"
Sep 17 15:43:54.097592 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:54.097577    2555 scope.go:117] "RemoveContainer" containerID="3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342"
Sep 17 15:43:54.097891 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:43:54.097864    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342\": container with ID starting with 3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342 not found: ID does not exist" containerID="3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342"
Sep 17 15:43:54.097984 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:54.097896    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342"} err="failed to get container status \"3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342\": rpc error: code = NotFound desc = could not find container \"3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342\": container with ID starting with 3a5740cbeae816198586bd5ccf722ec6fa87118fe01bb1de3c4f0f04c5fbd342 not found: ID does not exist"
Sep 17 15:43:54.110382 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:54.110361    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64945d586d-b4gxt"]
Sep 17 15:43:54.114128 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:54.114108    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64945d586d-b4gxt"]
Sep 17 15:43:54.693621 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:43:54.693595    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa6f84fb-c69f-4c1d-967d-ee53355f545c" path="/var/lib/kubelet/pods/aa6f84fb-c69f-4c1d-967d-ee53355f545c/volumes"
Sep 17 15:45:01.844165 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.844128    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"]
Sep 17 15:45:01.844564 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.844366    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6f84fb-c69f-4c1d-967d-ee53355f545c" containerName="console"
Sep 17 15:45:01.844564 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.844378    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6f84fb-c69f-4c1d-967d-ee53355f545c" containerName="console"
Sep 17 15:45:01.844564 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.844439    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6f84fb-c69f-4c1d-967d-ee53355f545c" containerName="console"
Sep 17 15:45:01.846307 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.846290    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:01.851057 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.851035    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6hcns\""
Sep 17 15:45:01.851326 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.851300    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 15:45:01.851326 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.851306    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 15:45:01.859369 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.859348    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"]
Sep 17 15:45:01.915571 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.915544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:01.915678 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.915576    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:01.915728 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:01.915675    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2b67\" (UniqueName: \"kubernetes.io/projected/d21fb09d-acf6-482d-9177-e15de7f20d1a-kube-api-access-f2b67\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:02.016935 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:02.016911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2b67\" (UniqueName: \"kubernetes.io/projected/d21fb09d-acf6-482d-9177-e15de7f20d1a-kube-api-access-f2b67\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:02.017046 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:02.016945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:02.017046 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:02.016964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:02.017377 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:02.017344    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:02.017416 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:02.017362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:02.025125 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:02.025097    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2b67\" (UniqueName: \"kubernetes.io/projected/d21fb09d-acf6-482d-9177-e15de7f20d1a-kube-api-access-f2b67\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:02.155266 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:02.155213    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:02.421958 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:02.421892    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"]
Sep 17 15:45:02.424567 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:45:02.424537    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21fb09d_acf6_482d_9177_e15de7f20d1a.slice/crio-16b7f03e1451d518d04e393b9e431f8377da82e06de7c68bd124fdcbb038ffaa WatchSource:0}: Error finding container 16b7f03e1451d518d04e393b9e431f8377da82e06de7c68bd124fdcbb038ffaa: Status 404 returned error can't find the container with id 16b7f03e1451d518d04e393b9e431f8377da82e06de7c68bd124fdcbb038ffaa
Sep 17 15:45:02.426410 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:02.426393    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:45:03.330505 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:03.330469    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c" event={"ID":"d21fb09d-acf6-482d-9177-e15de7f20d1a","Type":"ContainerStarted","Data":"16b7f03e1451d518d04e393b9e431f8377da82e06de7c68bd124fdcbb038ffaa"}
Sep 17 15:45:08.349707 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:08.349671    2555 generic.go:356] "Generic (PLEG): container finished" podID="d21fb09d-acf6-482d-9177-e15de7f20d1a" containerID="2b3c166ba014940d61f39d5e3f829da30136c298b806416d440e30fee4bad848" exitCode=0
Sep 17 15:45:08.350136 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:08.349720    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c" event={"ID":"d21fb09d-acf6-482d-9177-e15de7f20d1a","Type":"ContainerDied","Data":"2b3c166ba014940d61f39d5e3f829da30136c298b806416d440e30fee4bad848"}
Sep 17 15:45:13.368763 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:13.368731    2555 generic.go:356] "Generic (PLEG): container finished" podID="d21fb09d-acf6-482d-9177-e15de7f20d1a" containerID="7440a142b5d5c7ae3fa7d34d9f86d1cc0ddd48944068cda38dfd90bdbd1d8469" exitCode=0
Sep 17 15:45:13.369133 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:13.368774    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c" event={"ID":"d21fb09d-acf6-482d-9177-e15de7f20d1a","Type":"ContainerDied","Data":"7440a142b5d5c7ae3fa7d34d9f86d1cc0ddd48944068cda38dfd90bdbd1d8469"}
Sep 17 15:45:19.395380 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:19.395349    2555 generic.go:356] "Generic (PLEG): container finished" podID="d21fb09d-acf6-482d-9177-e15de7f20d1a" containerID="e7000a7d5292f02a54b05019e6bfd1322012e23b0b86de3a416efaf870372136" exitCode=0
Sep 17 15:45:19.395771 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:19.395413    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c" event={"ID":"d21fb09d-acf6-482d-9177-e15de7f20d1a","Type":"ContainerDied","Data":"e7000a7d5292f02a54b05019e6bfd1322012e23b0b86de3a416efaf870372136"}
Sep 17 15:45:20.556149 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:20.556124    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:20.653959 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:20.653904    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-util\") pod \"d21fb09d-acf6-482d-9177-e15de7f20d1a\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") "
Sep 17 15:45:20.653959 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:20.653950    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-bundle\") pod \"d21fb09d-acf6-482d-9177-e15de7f20d1a\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") "
Sep 17 15:45:20.654135 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:20.653970    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2b67\" (UniqueName: \"kubernetes.io/projected/d21fb09d-acf6-482d-9177-e15de7f20d1a-kube-api-access-f2b67\") pod \"d21fb09d-acf6-482d-9177-e15de7f20d1a\" (UID: \"d21fb09d-acf6-482d-9177-e15de7f20d1a\") "
Sep 17 15:45:20.654522 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:20.654489    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-bundle" (OuterVolumeSpecName: "bundle") pod "d21fb09d-acf6-482d-9177-e15de7f20d1a" (UID: "d21fb09d-acf6-482d-9177-e15de7f20d1a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:20.656011 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:20.655975    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21fb09d-acf6-482d-9177-e15de7f20d1a-kube-api-access-f2b67" (OuterVolumeSpecName: "kube-api-access-f2b67") pod "d21fb09d-acf6-482d-9177-e15de7f20d1a" (UID: "d21fb09d-acf6-482d-9177-e15de7f20d1a"). InnerVolumeSpecName "kube-api-access-f2b67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:45:20.658021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:20.657998    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-util" (OuterVolumeSpecName: "util") pod "d21fb09d-acf6-482d-9177-e15de7f20d1a" (UID: "d21fb09d-acf6-482d-9177-e15de7f20d1a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:20.754740 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:20.754719    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-util\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:45:20.754740 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:20.754737    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d21fb09d-acf6-482d-9177-e15de7f20d1a-bundle\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:45:20.754891 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:20.754746    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2b67\" (UniqueName: \"kubernetes.io/projected/d21fb09d-acf6-482d-9177-e15de7f20d1a-kube-api-access-f2b67\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:45:21.404741 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:21.404712    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c" event={"ID":"d21fb09d-acf6-482d-9177-e15de7f20d1a","Type":"ContainerDied","Data":"16b7f03e1451d518d04e393b9e431f8377da82e06de7c68bd124fdcbb038ffaa"}
Sep 17 15:45:21.404894 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:21.404761    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16b7f03e1451d518d04e393b9e431f8377da82e06de7c68bd124fdcbb038ffaa"
Sep 17 15:45:21.404894 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:21.404724    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alnp5c"
Sep 17 15:45:24.516970 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.516933    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"]
Sep 17 15:45:24.517417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.517180    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21fb09d-acf6-482d-9177-e15de7f20d1a" containerName="pull"
Sep 17 15:45:24.517417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.517193    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21fb09d-acf6-482d-9177-e15de7f20d1a" containerName="pull"
Sep 17 15:45:24.517417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.517222    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21fb09d-acf6-482d-9177-e15de7f20d1a" containerName="extract"
Sep 17 15:45:24.517417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.517229    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21fb09d-acf6-482d-9177-e15de7f20d1a" containerName="extract"
Sep 17 15:45:24.517417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.517236    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21fb09d-acf6-482d-9177-e15de7f20d1a" containerName="util"
Sep 17 15:45:24.517417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.517241    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21fb09d-acf6-482d-9177-e15de7f20d1a" containerName="util"
Sep 17 15:45:24.517417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.517281    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21fb09d-acf6-482d-9177-e15de7f20d1a" containerName="extract"
Sep 17 15:45:24.519176 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.519160    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"
Sep 17 15:45:24.522441 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.522420    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 15:45:24.523282 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.523265    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 15:45:24.529892 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.529879    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 15:45:24.538334 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.538310    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"]
Sep 17 15:45:24.539395 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.539377    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-tgbxv\""
Sep 17 15:45:24.581882 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.581858    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ck44\" (UniqueName: \"kubernetes.io/projected/07091104-84d3-4ac0-ad87-f639e487c453-kube-api-access-7ck44\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t6cpx\" (UID: \"07091104-84d3-4ac0-ad87-f639e487c453\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"
Sep 17 15:45:24.581965 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.581900    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/07091104-84d3-4ac0-ad87-f639e487c453-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t6cpx\" (UID: \"07091104-84d3-4ac0-ad87-f639e487c453\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"
Sep 17 15:45:24.682490 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.682466    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ck44\" (UniqueName: \"kubernetes.io/projected/07091104-84d3-4ac0-ad87-f639e487c453-kube-api-access-7ck44\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t6cpx\" (UID: \"07091104-84d3-4ac0-ad87-f639e487c453\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"
Sep 17 15:45:24.682603 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.682504    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/07091104-84d3-4ac0-ad87-f639e487c453-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t6cpx\" (UID: \"07091104-84d3-4ac0-ad87-f639e487c453\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"
Sep 17 15:45:24.684699 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.684679    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/07091104-84d3-4ac0-ad87-f639e487c453-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t6cpx\" (UID: \"07091104-84d3-4ac0-ad87-f639e487c453\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"
Sep 17 15:45:24.692701 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.692683    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ck44\" (UniqueName: \"kubernetes.io/projected/07091104-84d3-4ac0-ad87-f639e487c453-kube-api-access-7ck44\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t6cpx\" (UID: \"07091104-84d3-4ac0-ad87-f639e487c453\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"
Sep 17 15:45:24.828575 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:24.828542    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"
Sep 17 15:45:25.170956 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:25.170931    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"]
Sep 17 15:45:25.173018 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:45:25.172990    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07091104_84d3_4ac0_ad87_f639e487c453.slice/crio-c51d30187e57ea93213ebaf5b2bdf366bfc48a59eaf8ea7ac551d2835ac701c0 WatchSource:0}: Error finding container c51d30187e57ea93213ebaf5b2bdf366bfc48a59eaf8ea7ac551d2835ac701c0: Status 404 returned error can't find the container with id c51d30187e57ea93213ebaf5b2bdf366bfc48a59eaf8ea7ac551d2835ac701c0
Sep 17 15:45:25.419202 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:25.419170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx" event={"ID":"07091104-84d3-4ac0-ad87-f639e487c453","Type":"ContainerStarted","Data":"c51d30187e57ea93213ebaf5b2bdf366bfc48a59eaf8ea7ac551d2835ac701c0"}
Sep 17 15:45:28.431842 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.431786    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx" event={"ID":"07091104-84d3-4ac0-ad87-f639e487c453","Type":"ContainerStarted","Data":"1c76f3109cf96ec5dfa1355b4cb9d0951888bf9bc060edfc0475e451355f03f7"}
Sep 17 15:45:28.432292 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.432018    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"
Sep 17 15:45:28.456377 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.456333    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx" podStartSLOduration=1.683801054 podStartE2EDuration="4.45631439s" totalImagesPullingTime="2.772513336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:28.454104844 +0000 UTC m=+410.344965388" watchObservedRunningTime="2026-09-17 15:45:28.45631439 +0000 UTC m=+410.347174940"
Sep 17 15:45:28.747338 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.747266    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-8lkrs"]
Sep 17 15:45:28.749133 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.749119    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:28.752956 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.752933    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xhfnh\""
Sep 17 15:45:28.753245 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.753229    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 15:45:28.753293 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.753244    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 15:45:28.761678 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.761657    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-8lkrs"]
Sep 17 15:45:28.814193 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.814169    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/dd09eb71-839b-4632-85fe-7eb459f6e4e3-cabundle0\") pod \"keda-operator-579c8958f7-8lkrs\" (UID: \"dd09eb71-839b-4632-85fe-7eb459f6e4e3\") " pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:28.814309 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.814206    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dd09eb71-839b-4632-85fe-7eb459f6e4e3-certificates\") pod \"keda-operator-579c8958f7-8lkrs\" (UID: \"dd09eb71-839b-4632-85fe-7eb459f6e4e3\") " pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:28.814309 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.814234    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmlj\" (UniqueName: \"kubernetes.io/projected/dd09eb71-839b-4632-85fe-7eb459f6e4e3-kube-api-access-9cmlj\") pod \"keda-operator-579c8958f7-8lkrs\" (UID: \"dd09eb71-839b-4632-85fe-7eb459f6e4e3\") " pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:28.915294 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.915265    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dd09eb71-839b-4632-85fe-7eb459f6e4e3-certificates\") pod \"keda-operator-579c8958f7-8lkrs\" (UID: \"dd09eb71-839b-4632-85fe-7eb459f6e4e3\") " pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:28.915434 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.915302    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmlj\" (UniqueName: \"kubernetes.io/projected/dd09eb71-839b-4632-85fe-7eb459f6e4e3-kube-api-access-9cmlj\") pod \"keda-operator-579c8958f7-8lkrs\" (UID: \"dd09eb71-839b-4632-85fe-7eb459f6e4e3\") " pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:28.915434 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.915331    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/dd09eb71-839b-4632-85fe-7eb459f6e4e3-cabundle0\") pod \"keda-operator-579c8958f7-8lkrs\" (UID: \"dd09eb71-839b-4632-85fe-7eb459f6e4e3\") " pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:28.915434 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:45:28.915411    2555 secret.go:281] references non-existent secret key: ca.crt
Sep 17 15:45:28.915434 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:45:28.915428    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 15:45:28.915434 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:45:28.915436    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-8lkrs: references non-existent secret key: ca.crt
Sep 17 15:45:28.915669 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:45:28.915490    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd09eb71-839b-4632-85fe-7eb459f6e4e3-certificates podName:dd09eb71-839b-4632-85fe-7eb459f6e4e3 nodeName:}" failed. No retries permitted until 2026-09-17 15:45:29.415473804 +0000 UTC m=+411.306334325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/dd09eb71-839b-4632-85fe-7eb459f6e4e3-certificates") pod "keda-operator-579c8958f7-8lkrs" (UID: "dd09eb71-839b-4632-85fe-7eb459f6e4e3") : references non-existent secret key: ca.crt
Sep 17 15:45:28.915941 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.915925    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/dd09eb71-839b-4632-85fe-7eb459f6e4e3-cabundle0\") pod \"keda-operator-579c8958f7-8lkrs\" (UID: \"dd09eb71-839b-4632-85fe-7eb459f6e4e3\") " pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:28.924675 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:28.924657    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmlj\" (UniqueName: \"kubernetes.io/projected/dd09eb71-839b-4632-85fe-7eb459f6e4e3-kube-api-access-9cmlj\") pod \"keda-operator-579c8958f7-8lkrs\" (UID: \"dd09eb71-839b-4632-85fe-7eb459f6e4e3\") " pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:29.418570 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:29.418534    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dd09eb71-839b-4632-85fe-7eb459f6e4e3-certificates\") pod \"keda-operator-579c8958f7-8lkrs\" (UID: \"dd09eb71-839b-4632-85fe-7eb459f6e4e3\") " pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:29.420917 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:29.420882    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dd09eb71-839b-4632-85fe-7eb459f6e4e3-certificates\") pod \"keda-operator-579c8958f7-8lkrs\" (UID: \"dd09eb71-839b-4632-85fe-7eb459f6e4e3\") " pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:29.658391 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:29.658360    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:29.925743 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:29.925716    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-8lkrs"]
Sep 17 15:45:29.928063 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:45:29.928036    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd09eb71_839b_4632_85fe_7eb459f6e4e3.slice/crio-483988279f8eff86ad84ec55c5828f0edf531fd0fd6ccb160b730f0afcb28c26 WatchSource:0}: Error finding container 483988279f8eff86ad84ec55c5828f0edf531fd0fd6ccb160b730f0afcb28c26: Status 404 returned error can't find the container with id 483988279f8eff86ad84ec55c5828f0edf531fd0fd6ccb160b730f0afcb28c26
Sep 17 15:45:30.443961 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:30.443920    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-8lkrs" event={"ID":"dd09eb71-839b-4632-85fe-7eb459f6e4e3","Type":"ContainerStarted","Data":"483988279f8eff86ad84ec55c5828f0edf531fd0fd6ccb160b730f0afcb28c26"}
Sep 17 15:45:33.458472 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:33.458438    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-8lkrs" event={"ID":"dd09eb71-839b-4632-85fe-7eb459f6e4e3","Type":"ContainerStarted","Data":"05e6255e6e8cd3f60030bb5478e888a442abdaaeb9b92356f9776bad3a73cf55"}
Sep 17 15:45:33.458930 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:33.458545    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:45:33.477785 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:33.477741    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-8lkrs" podStartSLOduration=2.578866321 podStartE2EDuration="5.477729189s" totalImagesPullingTime="2.898862868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:33.477012713 +0000 UTC m=+415.367873257" watchObservedRunningTime="2026-09-17 15:45:33.477729189 +0000 UTC m=+415.368589746"
Sep 17 15:45:49.437739 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:49.437710    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t6cpx"
Sep 17 15:45:54.464259 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:45:54.464230    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-8lkrs"
Sep 17 15:46:00.000139 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:00.000108    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbc8b4559-m5qpc"]
Sep 17 15:46:21.929172 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:21.929138    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"]
Sep 17 15:46:21.933795 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:21.933771    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:21.937947 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:21.937923    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 15:46:21.938058 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:21.937947    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6hcns\""
Sep 17 15:46:21.939017 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:21.939002    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 15:46:21.944100 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:21.944077    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"]
Sep 17 15:46:21.969240 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:21.969214    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx9h4\" (UniqueName: \"kubernetes.io/projected/bd3d73db-5ea1-43d1-b451-3867ac364471-kube-api-access-xx9h4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:21.969347 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:21.969249    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:21.969347 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:21.969268    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:22.069511 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.069486    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:22.069592 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.069513    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:22.069592 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.069578    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx9h4\" (UniqueName: \"kubernetes.io/projected/bd3d73db-5ea1-43d1-b451-3867ac364471-kube-api-access-xx9h4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:22.069931 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.069911    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:22.069971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.069930    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:22.078530 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.078511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx9h4\" (UniqueName: \"kubernetes.io/projected/bd3d73db-5ea1-43d1-b451-3867ac364471-kube-api-access-xx9h4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:22.243689 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.243620    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:22.514614 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.514590    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"]
Sep 17 15:46:22.516087 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:46:22.516061    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3d73db_5ea1_43d1_b451_3867ac364471.slice/crio-c77921735cd81c87ce729924bb8c8baee78cc06b9fdcc6159c0fb783adfddaef WatchSource:0}: Error finding container c77921735cd81c87ce729924bb8c8baee78cc06b9fdcc6159c0fb783adfddaef: Status 404 returned error can't find the container with id c77921735cd81c87ce729924bb8c8baee78cc06b9fdcc6159c0fb783adfddaef
Sep 17 15:46:22.638414 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.638384    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd3d73db-5ea1-43d1-b451-3867ac364471" containerID="cfb3aab8ff90796b1cf3c3255d7675b4b03972af46281f4a5c967354e75e317a" exitCode=0
Sep 17 15:46:22.638538 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.638418    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k" event={"ID":"bd3d73db-5ea1-43d1-b451-3867ac364471","Type":"ContainerDied","Data":"cfb3aab8ff90796b1cf3c3255d7675b4b03972af46281f4a5c967354e75e317a"}
Sep 17 15:46:22.638538 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:22.638451    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k" event={"ID":"bd3d73db-5ea1-43d1-b451-3867ac364471","Type":"ContainerStarted","Data":"c77921735cd81c87ce729924bb8c8baee78cc06b9fdcc6159c0fb783adfddaef"}
Sep 17 15:46:25.018356 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.018266    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fbc8b4559-m5qpc" podUID="a4977f4a-30f4-42c4-a42b-700523f4da39" containerName="console" containerID="cri-o://b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91" gracePeriod=15
Sep 17 15:46:25.251400 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.251377    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:46:25.295543 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.295485    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-oauth-serving-cert\") pod \"a4977f4a-30f4-42c4-a42b-700523f4da39\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") "
Sep 17 15:46:25.295543 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.295519    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-service-ca\") pod \"a4977f4a-30f4-42c4-a42b-700523f4da39\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") "
Sep 17 15:46:25.295707 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.295557    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-oauth-config\") pod \"a4977f4a-30f4-42c4-a42b-700523f4da39\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") "
Sep 17 15:46:25.295707 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.295612    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-serving-cert\") pod \"a4977f4a-30f4-42c4-a42b-700523f4da39\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") "
Sep 17 15:46:25.295707 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.295642    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-trusted-ca-bundle\") pod \"a4977f4a-30f4-42c4-a42b-700523f4da39\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") "
Sep 17 15:46:25.295815 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.295766    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-console-config\") pod \"a4977f4a-30f4-42c4-a42b-700523f4da39\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") "
Sep 17 15:46:25.295891 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.295816    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht4b2\" (UniqueName: \"kubernetes.io/projected/a4977f4a-30f4-42c4-a42b-700523f4da39-kube-api-access-ht4b2\") pod \"a4977f4a-30f4-42c4-a42b-700523f4da39\" (UID: \"a4977f4a-30f4-42c4-a42b-700523f4da39\") "
Sep 17 15:46:25.295953 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.295902    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a4977f4a-30f4-42c4-a42b-700523f4da39" (UID: "a4977f4a-30f4-42c4-a42b-700523f4da39"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:46:25.296007 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.295953    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-service-ca" (OuterVolumeSpecName: "service-ca") pod "a4977f4a-30f4-42c4-a42b-700523f4da39" (UID: "a4977f4a-30f4-42c4-a42b-700523f4da39"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:46:25.296139 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.296116    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a4977f4a-30f4-42c4-a42b-700523f4da39" (UID: "a4977f4a-30f4-42c4-a42b-700523f4da39"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:46:25.296248 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.296177    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-console-config" (OuterVolumeSpecName: "console-config") pod "a4977f4a-30f4-42c4-a42b-700523f4da39" (UID: "a4977f4a-30f4-42c4-a42b-700523f4da39"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:46:25.296248 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.296199    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-trusted-ca-bundle\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:25.296248 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.296243    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-oauth-serving-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:25.296350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.296260    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-service-ca\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:25.297753 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.297733    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a4977f4a-30f4-42c4-a42b-700523f4da39" (UID: "a4977f4a-30f4-42c4-a42b-700523f4da39"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:46:25.297816 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.297772    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a4977f4a-30f4-42c4-a42b-700523f4da39" (UID: "a4977f4a-30f4-42c4-a42b-700523f4da39"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:46:25.297873 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.297817    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4977f4a-30f4-42c4-a42b-700523f4da39-kube-api-access-ht4b2" (OuterVolumeSpecName: "kube-api-access-ht4b2") pod "a4977f4a-30f4-42c4-a42b-700523f4da39" (UID: "a4977f4a-30f4-42c4-a42b-700523f4da39"). InnerVolumeSpecName "kube-api-access-ht4b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:46:25.396857 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.396814    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-oauth-config\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:25.396857 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.396850    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4977f4a-30f4-42c4-a42b-700523f4da39-console-serving-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:25.396857 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.396859    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4977f4a-30f4-42c4-a42b-700523f4da39-console-config\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:25.397060 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.396868    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht4b2\" (UniqueName: \"kubernetes.io/projected/a4977f4a-30f4-42c4-a42b-700523f4da39-kube-api-access-ht4b2\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:25.652035 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.652000    2555 generic.go:356] "Generic (PLEG): container finished" podID="a4977f4a-30f4-42c4-a42b-700523f4da39" containerID="b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91" exitCode=0
Sep 17 15:46:25.652035 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.652039    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbc8b4559-m5qpc"
Sep 17 15:46:25.652254 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.652090    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbc8b4559-m5qpc" event={"ID":"a4977f4a-30f4-42c4-a42b-700523f4da39","Type":"ContainerDied","Data":"b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91"}
Sep 17 15:46:25.652254 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.652137    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbc8b4559-m5qpc" event={"ID":"a4977f4a-30f4-42c4-a42b-700523f4da39","Type":"ContainerDied","Data":"ffa508ce6842e66edda7bd54fd81e24585ab2361db058608a8bc314c2885d998"}
Sep 17 15:46:25.652254 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.652165    2555 scope.go:117] "RemoveContainer" containerID="b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91"
Sep 17 15:46:25.657987 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.657876    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd3d73db-5ea1-43d1-b451-3867ac364471" containerID="43c8c82961ef7118759975a93a5ba2aab8a390dfe945d8bab4fad7f2cdffbc43" exitCode=0
Sep 17 15:46:25.657987 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.657925    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k" event={"ID":"bd3d73db-5ea1-43d1-b451-3867ac364471","Type":"ContainerDied","Data":"43c8c82961ef7118759975a93a5ba2aab8a390dfe945d8bab4fad7f2cdffbc43"}
Sep 17 15:46:25.665640 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.665619    2555 scope.go:117] "RemoveContainer" containerID="b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91"
Sep 17 15:46:25.665951 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:46:25.665925    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91\": container with ID starting with b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91 not found: ID does not exist" containerID="b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91"
Sep 17 15:46:25.666025 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.665961    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91"} err="failed to get container status \"b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91\": rpc error: code = NotFound desc = could not find container \"b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91\": container with ID starting with b23806dae9b87fa431bbe3c98f38b4ed4406ac72bb84445cfc4f77153a29ff91 not found: ID does not exist"
Sep 17 15:46:25.691232 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.691204    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbc8b4559-m5qpc"]
Sep 17 15:46:25.693504 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:25.693484    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fbc8b4559-m5qpc"]
Sep 17 15:46:26.665336 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:26.665301    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd3d73db-5ea1-43d1-b451-3867ac364471" containerID="c8cbe5e92944850c70e1af9b8114fa34bd5be1bed136c6ab1b4ef388f5b3ee09" exitCode=0
Sep 17 15:46:26.665336 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:26.665341    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k" event={"ID":"bd3d73db-5ea1-43d1-b451-3867ac364471","Type":"ContainerDied","Data":"c8cbe5e92944850c70e1af9b8114fa34bd5be1bed136c6ab1b4ef388f5b3ee09"}
Sep 17 15:46:26.692680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:26.692654    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4977f4a-30f4-42c4-a42b-700523f4da39" path="/var/lib/kubelet/pods/a4977f4a-30f4-42c4-a42b-700523f4da39/volumes"
Sep 17 15:46:27.789134 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:27.789114    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:27.814144 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:27.814121    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-util\") pod \"bd3d73db-5ea1-43d1-b451-3867ac364471\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") "
Sep 17 15:46:27.814249 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:27.814149    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-bundle\") pod \"bd3d73db-5ea1-43d1-b451-3867ac364471\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") "
Sep 17 15:46:27.814249 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:27.814187    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx9h4\" (UniqueName: \"kubernetes.io/projected/bd3d73db-5ea1-43d1-b451-3867ac364471-kube-api-access-xx9h4\") pod \"bd3d73db-5ea1-43d1-b451-3867ac364471\" (UID: \"bd3d73db-5ea1-43d1-b451-3867ac364471\") "
Sep 17 15:46:27.815145 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:27.815116    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-bundle" (OuterVolumeSpecName: "bundle") pod "bd3d73db-5ea1-43d1-b451-3867ac364471" (UID: "bd3d73db-5ea1-43d1-b451-3867ac364471"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:46:27.816162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:27.816130    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3d73db-5ea1-43d1-b451-3867ac364471-kube-api-access-xx9h4" (OuterVolumeSpecName: "kube-api-access-xx9h4") pod "bd3d73db-5ea1-43d1-b451-3867ac364471" (UID: "bd3d73db-5ea1-43d1-b451-3867ac364471"). InnerVolumeSpecName "kube-api-access-xx9h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:46:27.888802 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:27.888780    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-util" (OuterVolumeSpecName: "util") pod "bd3d73db-5ea1-43d1-b451-3867ac364471" (UID: "bd3d73db-5ea1-43d1-b451-3867ac364471"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:46:27.914698 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:27.914671    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-util\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:27.914698 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:27.914692    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd3d73db-5ea1-43d1-b451-3867ac364471-bundle\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:27.914816 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:27.914701    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx9h4\" (UniqueName: \"kubernetes.io/projected/bd3d73db-5ea1-43d1-b451-3867ac364471-kube-api-access-xx9h4\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:28.674546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:28.674507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k" event={"ID":"bd3d73db-5ea1-43d1-b451-3867ac364471","Type":"ContainerDied","Data":"c77921735cd81c87ce729924bb8c8baee78cc06b9fdcc6159c0fb783adfddaef"}
Sep 17 15:46:28.674698 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:28.674559    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c77921735cd81c87ce729924bb8c8baee78cc06b9fdcc6159c0fb783adfddaef"
Sep 17 15:46:28.674698 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:28.674528    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404jw6k"
Sep 17 15:46:34.797604 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797562    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"]
Sep 17 15:46:34.798006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797808    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3d73db-5ea1-43d1-b451-3867ac364471" containerName="pull"
Sep 17 15:46:34.798006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797819    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3d73db-5ea1-43d1-b451-3867ac364471" containerName="pull"
Sep 17 15:46:34.798006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797844    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4977f4a-30f4-42c4-a42b-700523f4da39" containerName="console"
Sep 17 15:46:34.798006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797850    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4977f4a-30f4-42c4-a42b-700523f4da39" containerName="console"
Sep 17 15:46:34.798006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797860    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3d73db-5ea1-43d1-b451-3867ac364471" containerName="extract"
Sep 17 15:46:34.798006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797866    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3d73db-5ea1-43d1-b451-3867ac364471" containerName="extract"
Sep 17 15:46:34.798006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797872    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3d73db-5ea1-43d1-b451-3867ac364471" containerName="util"
Sep 17 15:46:34.798006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797878    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3d73db-5ea1-43d1-b451-3867ac364471" containerName="util"
Sep 17 15:46:34.798006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797922    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4977f4a-30f4-42c4-a42b-700523f4da39" containerName="console"
Sep 17 15:46:34.798006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.797931    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3d73db-5ea1-43d1-b451-3867ac364471" containerName="extract"
Sep 17 15:46:34.801538 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.801519    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"
Sep 17 15:46:34.804195 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.804168    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:46:34.804291 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.804209    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 15:46:34.804291 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.804168    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vd6vd\""
Sep 17 15:46:34.812222 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.812203    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"]
Sep 17 15:46:34.866259 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.866233    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2djhj\" (UniqueName: \"kubernetes.io/projected/2a3742d2-ff22-44d3-a6eb-fb2de7c085d4-kube-api-access-2djhj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-xb9wq\" (UID: \"2a3742d2-ff22-44d3-a6eb-fb2de7c085d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"
Sep 17 15:46:34.866351 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.866268    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a3742d2-ff22-44d3-a6eb-fb2de7c085d4-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-xb9wq\" (UID: \"2a3742d2-ff22-44d3-a6eb-fb2de7c085d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"
Sep 17 15:46:34.967167 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.967138    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2djhj\" (UniqueName: \"kubernetes.io/projected/2a3742d2-ff22-44d3-a6eb-fb2de7c085d4-kube-api-access-2djhj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-xb9wq\" (UID: \"2a3742d2-ff22-44d3-a6eb-fb2de7c085d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"
Sep 17 15:46:34.967246 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.967179    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a3742d2-ff22-44d3-a6eb-fb2de7c085d4-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-xb9wq\" (UID: \"2a3742d2-ff22-44d3-a6eb-fb2de7c085d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"
Sep 17 15:46:34.967489 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.967472    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a3742d2-ff22-44d3-a6eb-fb2de7c085d4-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-xb9wq\" (UID: \"2a3742d2-ff22-44d3-a6eb-fb2de7c085d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"
Sep 17 15:46:34.974790 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:34.974772    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2djhj\" (UniqueName: \"kubernetes.io/projected/2a3742d2-ff22-44d3-a6eb-fb2de7c085d4-kube-api-access-2djhj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-xb9wq\" (UID: \"2a3742d2-ff22-44d3-a6eb-fb2de7c085d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"
Sep 17 15:46:35.110389 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:35.110371    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"
Sep 17 15:46:35.387107 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:35.387087    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq"]
Sep 17 15:46:35.388704 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:46:35.388678    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3742d2_ff22_44d3_a6eb_fb2de7c085d4.slice/crio-80275ab69ee97b41169eb0b00e3d00cd8786ced5ee879aa9a094a376e923a0cc WatchSource:0}: Error finding container 80275ab69ee97b41169eb0b00e3d00cd8786ced5ee879aa9a094a376e923a0cc: Status 404 returned error can't find the container with id 80275ab69ee97b41169eb0b00e3d00cd8786ced5ee879aa9a094a376e923a0cc
Sep 17 15:46:35.702860 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:35.702786    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq" event={"ID":"2a3742d2-ff22-44d3-a6eb-fb2de7c085d4","Type":"ContainerStarted","Data":"80275ab69ee97b41169eb0b00e3d00cd8786ced5ee879aa9a094a376e923a0cc"}
Sep 17 15:46:38.717195 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:38.717160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq" event={"ID":"2a3742d2-ff22-44d3-a6eb-fb2de7c085d4","Type":"ContainerStarted","Data":"64686cb63fc9ec1e36d536f861f16e359233fa6ede04d9f104917d2f14a54fad"}
Sep 17 15:46:38.739868 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:38.739793    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-xb9wq" podStartSLOduration=2.233957914 podStartE2EDuration="4.739775679s" totalImagesPullingTime="2.505817765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:46:38.739470173 +0000 UTC m=+480.630330719" watchObservedRunningTime="2026-09-17 15:46:38.739775679 +0000 UTC m=+480.630636225"
Sep 17 15:46:42.569419 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.569385    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"]
Sep 17 15:46:42.572259 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.572244    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"
Sep 17 15:46:42.574786 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.574766    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 15:46:42.575882 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.575862    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tzrbn\""
Sep 17 15:46:42.575973 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.575870    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 15:46:42.581221 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.581198    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"]
Sep 17 15:46:42.626605 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.626586    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl55z\" (UniqueName: \"kubernetes.io/projected/81681687-cb69-466b-9e22-8b257469ea6d-kube-api-access-rl55z\") pod \"cert-manager-cainjector-559cdd8c87-4n2bk\" (UID: \"81681687-cb69-466b-9e22-8b257469ea6d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"
Sep 17 15:46:42.626707 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.626612    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81681687-cb69-466b-9e22-8b257469ea6d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4n2bk\" (UID: \"81681687-cb69-466b-9e22-8b257469ea6d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"
Sep 17 15:46:42.727433 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.727411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl55z\" (UniqueName: \"kubernetes.io/projected/81681687-cb69-466b-9e22-8b257469ea6d-kube-api-access-rl55z\") pod \"cert-manager-cainjector-559cdd8c87-4n2bk\" (UID: \"81681687-cb69-466b-9e22-8b257469ea6d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"
Sep 17 15:46:42.727539 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.727441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81681687-cb69-466b-9e22-8b257469ea6d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4n2bk\" (UID: \"81681687-cb69-466b-9e22-8b257469ea6d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"
Sep 17 15:46:42.737550 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.737522    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81681687-cb69-466b-9e22-8b257469ea6d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4n2bk\" (UID: \"81681687-cb69-466b-9e22-8b257469ea6d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"
Sep 17 15:46:42.737727 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.737709    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl55z\" (UniqueName: \"kubernetes.io/projected/81681687-cb69-466b-9e22-8b257469ea6d-kube-api-access-rl55z\") pod \"cert-manager-cainjector-559cdd8c87-4n2bk\" (UID: \"81681687-cb69-466b-9e22-8b257469ea6d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"
Sep 17 15:46:42.880651 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:42.880620    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"
Sep 17 15:46:43.174677 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:43.174483    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk"]
Sep 17 15:46:43.177431 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:46:43.177403    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81681687_cb69_466b_9e22_8b257469ea6d.slice/crio-7104ae188a902e02f7f346cc3ca95d1180e98919282a91260faa82e7ec40cf32 WatchSource:0}: Error finding container 7104ae188a902e02f7f346cc3ca95d1180e98919282a91260faa82e7ec40cf32: Status 404 returned error can't find the container with id 7104ae188a902e02f7f346cc3ca95d1180e98919282a91260faa82e7ec40cf32
Sep 17 15:46:43.737301 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:43.737269    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk" event={"ID":"81681687-cb69-466b-9e22-8b257469ea6d","Type":"ContainerStarted","Data":"7104ae188a902e02f7f346cc3ca95d1180e98919282a91260faa82e7ec40cf32"}
Sep 17 15:46:44.066910 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.066819    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"]
Sep 17 15:46:44.069514 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.069497    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.071932 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.071910    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 15:46:44.072026 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.071965    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6hcns\""
Sep 17 15:46:44.072938 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.072925    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 15:46:44.081735 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.081715    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"]
Sep 17 15:46:44.136100 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.136074    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5v7m\" (UniqueName: \"kubernetes.io/projected/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-kube-api-access-t5v7m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.136212 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.136105    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.136212 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.136145    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.236479 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.236453    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.236626 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.236525    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5v7m\" (UniqueName: \"kubernetes.io/projected/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-kube-api-access-t5v7m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.236626 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.236562    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.236876 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.236848    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.236954 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.236928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.245458 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.245439    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5v7m\" (UniqueName: \"kubernetes.io/projected/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-kube-api-access-t5v7m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.378380 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.378352    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:44.658378 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.658349    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"]
Sep 17 15:46:44.661108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:46:44.661081    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f83d3a_5f39_440c_bdb4_640ff45dc2b9.slice/crio-e115ed23958caa47e1130fb6d67eab68c314c3625f52f7816683395551b1a592 WatchSource:0}: Error finding container e115ed23958caa47e1130fb6d67eab68c314c3625f52f7816683395551b1a592: Status 404 returned error can't find the container with id e115ed23958caa47e1130fb6d67eab68c314c3625f52f7816683395551b1a592
Sep 17 15:46:44.744727 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.744688    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875" event={"ID":"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9","Type":"ContainerStarted","Data":"e5c064895dae3ad9a1ab9b1fbebbc4d0792d1ab7dc50941b3b3d1993b3c89b2a"}
Sep 17 15:46:44.745160 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:44.744747    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875" event={"ID":"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9","Type":"ContainerStarted","Data":"e115ed23958caa47e1130fb6d67eab68c314c3625f52f7816683395551b1a592"}
Sep 17 15:46:45.751022 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:45.750991    2555 generic.go:356] "Generic (PLEG): container finished" podID="a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" containerID="e5c064895dae3ad9a1ab9b1fbebbc4d0792d1ab7dc50941b3b3d1993b3c89b2a" exitCode=0
Sep 17 15:46:45.751426 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:45.751081    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875" event={"ID":"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9","Type":"ContainerDied","Data":"e5c064895dae3ad9a1ab9b1fbebbc4d0792d1ab7dc50941b3b3d1993b3c89b2a"}
Sep 17 15:46:46.757263 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:46.757215    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk" event={"ID":"81681687-cb69-466b-9e22-8b257469ea6d","Type":"ContainerStarted","Data":"21c15e516b41ea57d19a96d1a40a416753bed39c34e735bdbf1e4a302b3fa35e"}
Sep 17 15:46:46.774465 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:46.774363    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n2bk" podStartSLOduration=1.911857836 podStartE2EDuration="4.77434152s" totalImagesPullingTime="2.862483684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:46:46.772758766 +0000 UTC m=+488.663619311" watchObservedRunningTime="2026-09-17 15:46:46.77434152 +0000 UTC m=+488.665202066"
Sep 17 15:46:47.763467 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:47.763434    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875" event={"ID":"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9","Type":"ContainerStarted","Data":"98a8b80d7805f06b91e2df19ec3b7f7238eafa26b0bbfc4ee8336428823aad3c"}
Sep 17 15:46:48.768723 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:48.768689    2555 generic.go:356] "Generic (PLEG): container finished" podID="a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" containerID="98a8b80d7805f06b91e2df19ec3b7f7238eafa26b0bbfc4ee8336428823aad3c" exitCode=0
Sep 17 15:46:48.769107 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:48.768772    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875" event={"ID":"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9","Type":"ContainerDied","Data":"98a8b80d7805f06b91e2df19ec3b7f7238eafa26b0bbfc4ee8336428823aad3c"}
Sep 17 15:46:49.775691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:49.775654    2555 generic.go:356] "Generic (PLEG): container finished" podID="a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" containerID="ef18bf555ecc626f224b36b29b0cdd5e988958d65024f1f2c084b9163da44ef0" exitCode=0
Sep 17 15:46:49.776065 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:49.775744    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875" event={"ID":"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9","Type":"ContainerDied","Data":"ef18bf555ecc626f224b36b29b0cdd5e988958d65024f1f2c084b9163da44ef0"}
Sep 17 15:46:50.893459 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:50.893434    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:50.991409 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:50.991388    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-util\") pod \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") "
Sep 17 15:46:50.991515 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:50.991429    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5v7m\" (UniqueName: \"kubernetes.io/projected/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-kube-api-access-t5v7m\") pod \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") "
Sep 17 15:46:50.991515 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:50.991445    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-bundle\") pod \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\" (UID: \"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9\") "
Sep 17 15:46:50.991897 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:50.991852    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-bundle" (OuterVolumeSpecName: "bundle") pod "a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" (UID: "a7f83d3a-5f39-440c-bdb4-640ff45dc2b9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:46:50.993349 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:50.993329    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-kube-api-access-t5v7m" (OuterVolumeSpecName: "kube-api-access-t5v7m") pod "a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" (UID: "a7f83d3a-5f39-440c-bdb4-640ff45dc2b9"). InnerVolumeSpecName "kube-api-access-t5v7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:46:50.995866 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:50.995845    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-util" (OuterVolumeSpecName: "util") pod "a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" (UID: "a7f83d3a-5f39-440c-bdb4-640ff45dc2b9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:46:51.092435 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:51.092412    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-util\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:51.092519 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:51.092453    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5v7m\" (UniqueName: \"kubernetes.io/projected/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-kube-api-access-t5v7m\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:51.092519 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:51.092464    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7f83d3a-5f39-440c-bdb4-640ff45dc2b9-bundle\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:46:51.786366 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:51.786335    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875"
Sep 17 15:46:51.786519 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:51.786359    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnx875" event={"ID":"a7f83d3a-5f39-440c-bdb4-640ff45dc2b9","Type":"ContainerDied","Data":"e115ed23958caa47e1130fb6d67eab68c314c3625f52f7816683395551b1a592"}
Sep 17 15:46:51.786519 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:51.786406    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e115ed23958caa47e1130fb6d67eab68c314c3625f52f7816683395551b1a592"
Sep 17 15:46:57.292837 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.292789    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"]
Sep 17 15:46:57.293237 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.293050    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" containerName="util"
Sep 17 15:46:57.293237 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.293061    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" containerName="util"
Sep 17 15:46:57.293237 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.293069    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" containerName="extract"
Sep 17 15:46:57.293237 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.293074    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" containerName="extract"
Sep 17 15:46:57.293237 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.293091    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" containerName="pull"
Sep 17 15:46:57.293237 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.293096    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" containerName="pull"
Sep 17 15:46:57.293237 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.293143    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f83d3a-5f39-440c-bdb4-640ff45dc2b9" containerName="extract"
Sep 17 15:46:57.295608 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.295593    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"
Sep 17 15:46:57.299718 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.299698    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:46:57.300037 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.300021    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 15:46:57.300929 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.300913    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-77r89\""
Sep 17 15:46:57.311439 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.311414    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"]
Sep 17 15:46:57.434515 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.434495    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc48b0e-1935-4164-8bfa-069634107495-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jvkjc\" (UID: \"2dc48b0e-1935-4164-8bfa-069634107495\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"
Sep 17 15:46:57.434613 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.434544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrt2w\" (UniqueName: \"kubernetes.io/projected/2dc48b0e-1935-4164-8bfa-069634107495-kube-api-access-rrt2w\") pod \"openshift-lws-operator-fd8ccff4c-jvkjc\" (UID: \"2dc48b0e-1935-4164-8bfa-069634107495\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"
Sep 17 15:46:57.534933 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.534901    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrt2w\" (UniqueName: \"kubernetes.io/projected/2dc48b0e-1935-4164-8bfa-069634107495-kube-api-access-rrt2w\") pod \"openshift-lws-operator-fd8ccff4c-jvkjc\" (UID: \"2dc48b0e-1935-4164-8bfa-069634107495\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"
Sep 17 15:46:57.535021 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.534956    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc48b0e-1935-4164-8bfa-069634107495-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jvkjc\" (UID: \"2dc48b0e-1935-4164-8bfa-069634107495\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"
Sep 17 15:46:57.535251 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.535237    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc48b0e-1935-4164-8bfa-069634107495-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jvkjc\" (UID: \"2dc48b0e-1935-4164-8bfa-069634107495\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"
Sep 17 15:46:57.544147 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.544092    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrt2w\" (UniqueName: \"kubernetes.io/projected/2dc48b0e-1935-4164-8bfa-069634107495-kube-api-access-rrt2w\") pod \"openshift-lws-operator-fd8ccff4c-jvkjc\" (UID: \"2dc48b0e-1935-4164-8bfa-069634107495\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"
Sep 17 15:46:57.604208 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.604180    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"
Sep 17 15:46:57.889902 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:57.889872    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc"]
Sep 17 15:46:57.892749 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:46:57.892724    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dc48b0e_1935_4164_8bfa_069634107495.slice/crio-b82e303077bcbf7811f653a876c1ee07d94718f9968090608a2f4ca0da25122e WatchSource:0}: Error finding container b82e303077bcbf7811f653a876c1ee07d94718f9968090608a2f4ca0da25122e: Status 404 returned error can't find the container with id b82e303077bcbf7811f653a876c1ee07d94718f9968090608a2f4ca0da25122e
Sep 17 15:46:58.816177 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:46:58.816140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc" event={"ID":"2dc48b0e-1935-4164-8bfa-069634107495","Type":"ContainerStarted","Data":"b82e303077bcbf7811f653a876c1ee07d94718f9968090608a2f4ca0da25122e"}
Sep 17 15:47:00.826440 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:00.826400    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc" event={"ID":"2dc48b0e-1935-4164-8bfa-069634107495","Type":"ContainerStarted","Data":"2fc1c0a84e53d722beb28a998cf9839946ededb239d1e527ae0d545a662b8785"}
Sep 17 15:47:00.846251 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:00.846196    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvkjc" podStartSLOduration=1.5695199 podStartE2EDuration="3.846180849s" totalImagesPullingTime="2.276660949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:00.845021186 +0000 UTC m=+502.735881732" watchObservedRunningTime="2026-09-17 15:47:00.846180849 +0000 UTC m=+502.737041393"
Sep 17 15:47:19.755662 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.755632    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"]
Sep 17 15:47:19.757672 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.757643    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.760753 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.760735    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 15:47:19.761732 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.761679    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-8gdk4\""
Sep 17 15:47:19.761801 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.761725    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 15:47:19.761919 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.761811    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 15:47:19.778716 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.778697    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"]
Sep 17 15:47:19.888497 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.888472    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.888621 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.888502    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.888621 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.888523    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.888621 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.888576    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86b9\" (UniqueName: \"kubernetes.io/projected/7e9582a6-488e-486d-a7ab-20ad6fb18be2-kube-api-access-j86b9\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.888621 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.888618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.888752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.888660    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.888752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.888707    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.888752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.888727    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.888752 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.888741    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989043 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989018    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989043 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989045    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989208 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989067    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989208 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989083    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989208 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989352 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989230    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86b9\" (UniqueName: \"kubernetes.io/projected/7e9582a6-488e-486d-a7ab-20ad6fb18be2-kube-api-access-j86b9\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989352 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989352 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989326    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989503 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989364    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989503 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989455    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989503 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989750 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989915 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.989915 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.989874    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.991560 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.991541    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:19.991845 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:19.991813    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:20.001857 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:20.001837    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86b9\" (UniqueName: \"kubernetes.io/projected/7e9582a6-488e-486d-a7ab-20ad6fb18be2-kube-api-access-j86b9\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:20.004006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:20.003983    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9582a6-488e-486d-a7ab-20ad6fb18be2-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-crjlt\" (UID: \"7e9582a6-488e-486d-a7ab-20ad6fb18be2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:20.068680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:20.068626    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:20.363243 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:20.363215    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"]
Sep 17 15:47:20.366779 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:47:20.366754    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9582a6_488e_486d_a7ab_20ad6fb18be2.slice/crio-31f96edba7babcd8e7535629e105aa6426a33f8874934988e24e5f117fd73f67 WatchSource:0}: Error finding container 31f96edba7babcd8e7535629e105aa6426a33f8874934988e24e5f117fd73f67: Status 404 returned error can't find the container with id 31f96edba7babcd8e7535629e105aa6426a33f8874934988e24e5f117fd73f67
Sep 17 15:47:20.908597 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:20.908564    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt" event={"ID":"7e9582a6-488e-486d-a7ab-20ad6fb18be2","Type":"ContainerStarted","Data":"31f96edba7babcd8e7535629e105aa6426a33f8874934988e24e5f117fd73f67"}
Sep 17 15:47:23.483033 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:23.482987    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 15:47:23.483311 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:23.483070    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 15:47:23.483311 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:23.483098    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 15:47:23.924201 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:23.924161    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt" event={"ID":"7e9582a6-488e-486d-a7ab-20ad6fb18be2","Type":"ContainerStarted","Data":"2951c63bfdaa5b2ac6def4e023bb8798e52bb6d5de28ce411839ea3a38965f2d"}
Sep 17 15:47:23.947552 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:23.947504    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt" podStartSLOduration=1.833484682 podStartE2EDuration="4.947489322s" totalImagesPullingTime="3.11400464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:23.947042478 +0000 UTC m=+525.837903023" watchObservedRunningTime="2026-09-17 15:47:23.947489322 +0000 UTC m=+525.838349867"
Sep 17 15:47:24.069884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.069855    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:24.074094 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.074067    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:24.704241 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.704208    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"]
Sep 17 15:47:24.706418 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.706402    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.711762 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.711733    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 15:47:24.712970 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.712952    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-k22bg\""
Sep 17 15:47:24.713068 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.713006    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 15:47:24.713946 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.713929    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 15:47:24.728646 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.728617    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"]
Sep 17 15:47:24.825416 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.825379    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be93defc-ebcd-4c45-a09d-bfbc53de9f76-cert\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.825577 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.825433    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/be93defc-ebcd-4c45-a09d-bfbc53de9f76-manager-config\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.825577 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.825478    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/be93defc-ebcd-4c45-a09d-bfbc53de9f76-metrics-cert\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.825577 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.825510    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgbh6\" (UniqueName: \"kubernetes.io/projected/be93defc-ebcd-4c45-a09d-bfbc53de9f76-kube-api-access-xgbh6\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.926091 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.926053    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be93defc-ebcd-4c45-a09d-bfbc53de9f76-cert\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.926279 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.926098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/be93defc-ebcd-4c45-a09d-bfbc53de9f76-manager-config\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.926279 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.926149    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/be93defc-ebcd-4c45-a09d-bfbc53de9f76-metrics-cert\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.926279 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.926181    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgbh6\" (UniqueName: \"kubernetes.io/projected/be93defc-ebcd-4c45-a09d-bfbc53de9f76-kube-api-access-xgbh6\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.926844 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.926800    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/be93defc-ebcd-4c45-a09d-bfbc53de9f76-manager-config\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.928709 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.928683    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be93defc-ebcd-4c45-a09d-bfbc53de9f76-cert\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.928976 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.928961    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/be93defc-ebcd-4c45-a09d-bfbc53de9f76-metrics-cert\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:24.929938 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.929926    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:24.930904 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.930883    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:47:24.935279 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:24.935261    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgbh6\" (UniqueName: \"kubernetes.io/projected/be93defc-ebcd-4c45-a09d-bfbc53de9f76-kube-api-access-xgbh6\") pod \"lws-controller-manager-65f7746b69-pcjww\" (UID: \"be93defc-ebcd-4c45-a09d-bfbc53de9f76\") " pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:25.014950 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:25.014920    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:25.289284 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:25.289262    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"]
Sep 17 15:47:25.290923 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:47:25.290883    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe93defc_ebcd_4c45_a09d_bfbc53de9f76.slice/crio-051b8efd37207645e12b8bad2be8c621e62d900e1f317d4dc28414fd24124877 WatchSource:0}: Error finding container 051b8efd37207645e12b8bad2be8c621e62d900e1f317d4dc28414fd24124877: Status 404 returned error can't find the container with id 051b8efd37207645e12b8bad2be8c621e62d900e1f317d4dc28414fd24124877
Sep 17 15:47:25.935518 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:25.935484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww" event={"ID":"be93defc-ebcd-4c45-a09d-bfbc53de9f76","Type":"ContainerStarted","Data":"051b8efd37207645e12b8bad2be8c621e62d900e1f317d4dc28414fd24124877"}
Sep 17 15:47:26.942387 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:26.942351    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww" event={"ID":"be93defc-ebcd-4c45-a09d-bfbc53de9f76","Type":"ContainerStarted","Data":"9e324c6799e1e4dc964adcf0055606591ad8487d5f8b728783cc09908bcd049c"}
Sep 17 15:47:26.959917 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:26.959851    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww" podStartSLOduration=1.4595925269999999 podStartE2EDuration="2.959814945s" totalImagesPullingTime="1.500222418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:26.959068967 +0000 UTC m=+528.849929511" watchObservedRunningTime="2026-09-17 15:47:26.959814945 +0000 UTC m=+528.850675489"
Sep 17 15:47:27.947368 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:27.947332    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:47:29.435555 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.435516    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"]
Sep 17 15:47:29.438189 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.438167    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.440938 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.440916    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 15:47:29.441975 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.441952    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 15:47:29.442114 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.442094    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6hcns\""
Sep 17 15:47:29.448119 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.448095    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"]
Sep 17 15:47:29.521278 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.521245    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"]
Sep 17 15:47:29.524033 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.524017    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.532784 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.532763    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"]
Sep 17 15:47:29.560124 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.560101    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.560214 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.560134    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncx9l\" (UniqueName: \"kubernetes.io/projected/8583d851-3b5b-49e9-9cc3-a1495455585b-kube-api-access-ncx9l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.560214 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.560166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.632760 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.632735    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"]
Sep 17 15:47:29.635216 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.635201    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.644462 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.644440    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"]
Sep 17 15:47:29.660515 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.660495    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.660600 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.660533    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sq6q\" (UniqueName: \"kubernetes.io/projected/56c38fff-a1a2-4f59-b1f0-83327ea3061c-kube-api-access-6sq6q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.660600 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.660589    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.660795 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.660626    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.660795 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.660658    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncx9l\" (UniqueName: \"kubernetes.io/projected/8583d851-3b5b-49e9-9cc3-a1495455585b-kube-api-access-ncx9l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.660795 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.660684    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.661107 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.660931    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.661107 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.661044    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.669467 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.669442    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncx9l\" (UniqueName: \"kubernetes.io/projected/8583d851-3b5b-49e9-9cc3-a1495455585b-kube-api-access-ncx9l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.728544 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.728490    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"]
Sep 17 15:47:29.730933 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.730916    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:29.739780 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.739764    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"]
Sep 17 15:47:29.746979 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.746960    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:29.761315 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.761290    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.761416 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.761336    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.761416 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.761377    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdn5\" (UniqueName: \"kubernetes.io/projected/eba480c7-9c34-4804-a7a8-37491fee90ea-kube-api-access-zkdn5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.761539 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.761426    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.761539 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.761491    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sq6q\" (UniqueName: \"kubernetes.io/projected/56c38fff-a1a2-4f59-b1f0-83327ea3061c-kube-api-access-6sq6q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.761660 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.761551    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.761717 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.761699    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.761838 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.761811    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.769213 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.769193    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sq6q\" (UniqueName: \"kubernetes.io/projected/56c38fff-a1a2-4f59-b1f0-83327ea3061c-kube-api-access-6sq6q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.832565 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.832538    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:29.862888 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.862860    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:29.863022 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.862919    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:29.863022 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.862968    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fq8b\" (UniqueName: \"kubernetes.io/projected/21a37064-3cb5-4bc2-9aad-dc261c983967-kube-api-access-6fq8b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:29.863022 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.862999    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.863182 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.863031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdn5\" (UniqueName: \"kubernetes.io/projected/eba480c7-9c34-4804-a7a8-37491fee90ea-kube-api-access-zkdn5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.863182 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.863096    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.863416 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.863398    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.863487 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.863420    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.871691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.871668    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdn5\" (UniqueName: \"kubernetes.io/projected/eba480c7-9c34-4804-a7a8-37491fee90ea-kube-api-access-zkdn5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.943404 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.943375    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:29.964218 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.964104    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:29.964218 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.964158    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:29.964218 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.964218    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fq8b\" (UniqueName: \"kubernetes.io/projected/21a37064-3cb5-4bc2-9aad-dc261c983967-kube-api-access-6fq8b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:29.965353 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.965232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:29.965353 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.965304    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:29.975244 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:29.975220    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fq8b\" (UniqueName: \"kubernetes.io/projected/21a37064-3cb5-4bc2-9aad-dc261c983967-kube-api-access-6fq8b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:30.019355 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.019319    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"]
Sep 17 15:47:30.019717 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:47:30.019691    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8583d851_3b5b_49e9_9cc3_a1495455585b.slice/crio-959d94912d9c910ef9e53e46db1f949acc21e8b3d9d982db66c49c49f9967f4b WatchSource:0}: Error finding container 959d94912d9c910ef9e53e46db1f949acc21e8b3d9d982db66c49c49f9967f4b: Status 404 returned error can't find the container with id 959d94912d9c910ef9e53e46db1f949acc21e8b3d9d982db66c49c49f9967f4b
Sep 17 15:47:30.039730 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.039708    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:30.105901 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.105878    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"]
Sep 17 15:47:30.107544 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:47:30.107521    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c38fff_a1a2_4f59_b1f0_83327ea3061c.slice/crio-9a287acdac4ca517bbea21e0cf577f2bc4450bee50378b6b37798ccc7d4200ce WatchSource:0}: Error finding container 9a287acdac4ca517bbea21e0cf577f2bc4450bee50378b6b37798ccc7d4200ce: Status 404 returned error can't find the container with id 9a287acdac4ca517bbea21e0cf577f2bc4450bee50378b6b37798ccc7d4200ce
Sep 17 15:47:30.213384 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.213358    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"]
Sep 17 15:47:30.214622 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:47:30.214596    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba480c7_9c34_4804_a7a8_37491fee90ea.slice/crio-1e5291fc5c2ddc4377fbbd2a40b17e8ae92368b4da0cf9571fe93d3d6a2f7d88 WatchSource:0}: Error finding container 1e5291fc5c2ddc4377fbbd2a40b17e8ae92368b4da0cf9571fe93d3d6a2f7d88: Status 404 returned error can't find the container with id 1e5291fc5c2ddc4377fbbd2a40b17e8ae92368b4da0cf9571fe93d3d6a2f7d88
Sep 17 15:47:30.311126 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.311104    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"]
Sep 17 15:47:30.363506 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:47:30.363479    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a37064_3cb5_4bc2_9aad_dc261c983967.slice/crio-1b489b18272bd2a72547ba5bf8e99636ac30fdbc5416cc42c7c9c86ecb9dc085 WatchSource:0}: Error finding container 1b489b18272bd2a72547ba5bf8e99636ac30fdbc5416cc42c7c9c86ecb9dc085: Status 404 returned error can't find the container with id 1b489b18272bd2a72547ba5bf8e99636ac30fdbc5416cc42c7c9c86ecb9dc085
Sep 17 15:47:30.961541 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.961503    2555 generic.go:356] "Generic (PLEG): container finished" podID="21a37064-3cb5-4bc2-9aad-dc261c983967" containerID="dc38be4ed84182dceac35eac05016ec20a4aa85914dfaa3a6e4ec0e2982fa16d" exitCode=0
Sep 17 15:47:30.961965 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.961564    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst" event={"ID":"21a37064-3cb5-4bc2-9aad-dc261c983967","Type":"ContainerDied","Data":"dc38be4ed84182dceac35eac05016ec20a4aa85914dfaa3a6e4ec0e2982fa16d"}
Sep 17 15:47:30.961965 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.961622    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst" event={"ID":"21a37064-3cb5-4bc2-9aad-dc261c983967","Type":"ContainerStarted","Data":"1b489b18272bd2a72547ba5bf8e99636ac30fdbc5416cc42c7c9c86ecb9dc085"}
Sep 17 15:47:30.963210 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.963188    2555 generic.go:356] "Generic (PLEG): container finished" podID="56c38fff-a1a2-4f59-b1f0-83327ea3061c" containerID="71b2b25c609c1bcebf7eed83a3973e7c87670c757e40d06b8b99a211565b6117" exitCode=0
Sep 17 15:47:30.963296 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.963259    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th" event={"ID":"56c38fff-a1a2-4f59-b1f0-83327ea3061c","Type":"ContainerDied","Data":"71b2b25c609c1bcebf7eed83a3973e7c87670c757e40d06b8b99a211565b6117"}
Sep 17 15:47:30.963350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.963305    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th" event={"ID":"56c38fff-a1a2-4f59-b1f0-83327ea3061c","Type":"ContainerStarted","Data":"9a287acdac4ca517bbea21e0cf577f2bc4450bee50378b6b37798ccc7d4200ce"}
Sep 17 15:47:30.964958 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.964939    2555 generic.go:356] "Generic (PLEG): container finished" podID="8583d851-3b5b-49e9-9cc3-a1495455585b" containerID="344818e7f4afe3388016a01e60b29db2cb6c76e727c8f8301574fa969a8f078b" exitCode=0
Sep 17 15:47:30.965085 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.965063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6" event={"ID":"8583d851-3b5b-49e9-9cc3-a1495455585b","Type":"ContainerDied","Data":"344818e7f4afe3388016a01e60b29db2cb6c76e727c8f8301574fa969a8f078b"}
Sep 17 15:47:30.965153 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.965110    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6" event={"ID":"8583d851-3b5b-49e9-9cc3-a1495455585b","Type":"ContainerStarted","Data":"959d94912d9c910ef9e53e46db1f949acc21e8b3d9d982db66c49c49f9967f4b"}
Sep 17 15:47:30.966880 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.966861    2555 generic.go:356] "Generic (PLEG): container finished" podID="eba480c7-9c34-4804-a7a8-37491fee90ea" containerID="645805fface4ef8178752c6d20ac95ab63fdbfcff3a83c32f6974106db9b099e" exitCode=0
Sep 17 15:47:30.966950 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.966897    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5" event={"ID":"eba480c7-9c34-4804-a7a8-37491fee90ea","Type":"ContainerDied","Data":"645805fface4ef8178752c6d20ac95ab63fdbfcff3a83c32f6974106db9b099e"}
Sep 17 15:47:30.966950 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:30.966923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5" event={"ID":"eba480c7-9c34-4804-a7a8-37491fee90ea","Type":"ContainerStarted","Data":"1e5291fc5c2ddc4377fbbd2a40b17e8ae92368b4da0cf9571fe93d3d6a2f7d88"}
Sep 17 15:47:31.975768 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:31.975695    2555 generic.go:356] "Generic (PLEG): container finished" podID="56c38fff-a1a2-4f59-b1f0-83327ea3061c" containerID="a664571dc6233a73687599e8b34b9c93d1a5a2974e2f9733140fc1d52b345b8f" exitCode=0
Sep 17 15:47:31.976163 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:31.975769    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th" event={"ID":"56c38fff-a1a2-4f59-b1f0-83327ea3061c","Type":"ContainerDied","Data":"a664571dc6233a73687599e8b34b9c93d1a5a2974e2f9733140fc1d52b345b8f"}
Sep 17 15:47:32.982571 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:32.982502    2555 generic.go:356] "Generic (PLEG): container finished" podID="eba480c7-9c34-4804-a7a8-37491fee90ea" containerID="bec2f8a38a940cad5adf62132a281985a8f6678648faf16d0328798fdb686f5e" exitCode=0
Sep 17 15:47:32.982957 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:32.982595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5" event={"ID":"eba480c7-9c34-4804-a7a8-37491fee90ea","Type":"ContainerDied","Data":"bec2f8a38a940cad5adf62132a281985a8f6678648faf16d0328798fdb686f5e"}
Sep 17 15:47:32.984648 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:32.984625    2555 generic.go:356] "Generic (PLEG): container finished" podID="21a37064-3cb5-4bc2-9aad-dc261c983967" containerID="ba6fe128a7c72be6e67b0751dbf7495fe3afb679bea4032de03db7cd7c643b05" exitCode=0
Sep 17 15:47:32.984756 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:32.984711    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst" event={"ID":"21a37064-3cb5-4bc2-9aad-dc261c983967","Type":"ContainerDied","Data":"ba6fe128a7c72be6e67b0751dbf7495fe3afb679bea4032de03db7cd7c643b05"}
Sep 17 15:47:32.987078 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:32.987057    2555 generic.go:356] "Generic (PLEG): container finished" podID="56c38fff-a1a2-4f59-b1f0-83327ea3061c" containerID="0fc8637a0152a8b23daae8aca9d3b94adbda3235cb2841d48429ae6676e06cde" exitCode=0
Sep 17 15:47:32.987180 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:32.987122    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th" event={"ID":"56c38fff-a1a2-4f59-b1f0-83327ea3061c","Type":"ContainerDied","Data":"0fc8637a0152a8b23daae8aca9d3b94adbda3235cb2841d48429ae6676e06cde"}
Sep 17 15:47:32.989054 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:32.989032    2555 generic.go:356] "Generic (PLEG): container finished" podID="8583d851-3b5b-49e9-9cc3-a1495455585b" containerID="5757d7f6696f3a5ed19833310ff6ccce123915a4ab0d2a9281b706c2f2a784c6" exitCode=0
Sep 17 15:47:32.989158 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:32.989085    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6" event={"ID":"8583d851-3b5b-49e9-9cc3-a1495455585b","Type":"ContainerDied","Data":"5757d7f6696f3a5ed19833310ff6ccce123915a4ab0d2a9281b706c2f2a784c6"}
Sep 17 15:47:33.996135 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:33.996043    2555 generic.go:356] "Generic (PLEG): container finished" podID="21a37064-3cb5-4bc2-9aad-dc261c983967" containerID="d697f0840f1e4eea94e846b7a809d766f7dd248061c40da4f3d806bf3ab9718d" exitCode=0
Sep 17 15:47:33.996596 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:33.996132    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst" event={"ID":"21a37064-3cb5-4bc2-9aad-dc261c983967","Type":"ContainerDied","Data":"d697f0840f1e4eea94e846b7a809d766f7dd248061c40da4f3d806bf3ab9718d"}
Sep 17 15:47:33.998374 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:33.998346    2555 generic.go:356] "Generic (PLEG): container finished" podID="8583d851-3b5b-49e9-9cc3-a1495455585b" containerID="1d21e55229ca276639ca626a2251ec206b71b2ee2e30993b781b90ff3cd83cea" exitCode=0
Sep 17 15:47:33.998516 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:33.998431    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6" event={"ID":"8583d851-3b5b-49e9-9cc3-a1495455585b","Type":"ContainerDied","Data":"1d21e55229ca276639ca626a2251ec206b71b2ee2e30993b781b90ff3cd83cea"}
Sep 17 15:47:34.000558 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.000537    2555 generic.go:356] "Generic (PLEG): container finished" podID="eba480c7-9c34-4804-a7a8-37491fee90ea" containerID="40b6835820d0b527ead4fcd3cc1ac39b2d7a7fa4ed5ddcd89e964e44a47534ca" exitCode=0
Sep 17 15:47:34.000641 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.000600    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5" event={"ID":"eba480c7-9c34-4804-a7a8-37491fee90ea","Type":"ContainerDied","Data":"40b6835820d0b527ead4fcd3cc1ac39b2d7a7fa4ed5ddcd89e964e44a47534ca"}
Sep 17 15:47:34.126284 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.126261    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:34.294095 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.294036    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sq6q\" (UniqueName: \"kubernetes.io/projected/56c38fff-a1a2-4f59-b1f0-83327ea3061c-kube-api-access-6sq6q\") pod \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") "
Sep 17 15:47:34.294203 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.294095    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-bundle\") pod \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") "
Sep 17 15:47:34.294203 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.294131    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-util\") pod \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\" (UID: \"56c38fff-a1a2-4f59-b1f0-83327ea3061c\") "
Sep 17 15:47:34.294678 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.294648    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-bundle" (OuterVolumeSpecName: "bundle") pod "56c38fff-a1a2-4f59-b1f0-83327ea3061c" (UID: "56c38fff-a1a2-4f59-b1f0-83327ea3061c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:34.296233 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.296207    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c38fff-a1a2-4f59-b1f0-83327ea3061c-kube-api-access-6sq6q" (OuterVolumeSpecName: "kube-api-access-6sq6q") pod "56c38fff-a1a2-4f59-b1f0-83327ea3061c" (UID: "56c38fff-a1a2-4f59-b1f0-83327ea3061c"). InnerVolumeSpecName "kube-api-access-6sq6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:47:34.299570 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.299549    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-util" (OuterVolumeSpecName: "util") pod "56c38fff-a1a2-4f59-b1f0-83327ea3061c" (UID: "56c38fff-a1a2-4f59-b1f0-83327ea3061c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:34.394812 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.394788    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sq6q\" (UniqueName: \"kubernetes.io/projected/56c38fff-a1a2-4f59-b1f0-83327ea3061c-kube-api-access-6sq6q\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:34.394812 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.394810    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-bundle\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:34.394961 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:34.394818    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56c38fff-a1a2-4f59-b1f0-83327ea3061c-util\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.008687 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.008616    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th"
Sep 17 15:47:35.008687 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.008648    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kf4th" event={"ID":"56c38fff-a1a2-4f59-b1f0-83327ea3061c","Type":"ContainerDied","Data":"9a287acdac4ca517bbea21e0cf577f2bc4450bee50378b6b37798ccc7d4200ce"}
Sep 17 15:47:35.009172 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.008701    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a287acdac4ca517bbea21e0cf577f2bc4450bee50378b6b37798ccc7d4200ce"
Sep 17 15:47:35.149948 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.149921    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:35.192643 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.192618    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:35.195759 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.195740    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:35.301540 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.301480    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-bundle\") pod \"eba480c7-9c34-4804-a7a8-37491fee90ea\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") "
Sep 17 15:47:35.301540 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.301513    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fq8b\" (UniqueName: \"kubernetes.io/projected/21a37064-3cb5-4bc2-9aad-dc261c983967-kube-api-access-6fq8b\") pod \"21a37064-3cb5-4bc2-9aad-dc261c983967\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") "
Sep 17 15:47:35.301540 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.301539    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkdn5\" (UniqueName: \"kubernetes.io/projected/eba480c7-9c34-4804-a7a8-37491fee90ea-kube-api-access-zkdn5\") pod \"eba480c7-9c34-4804-a7a8-37491fee90ea\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") "
Sep 17 15:47:35.301749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.301570    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncx9l\" (UniqueName: \"kubernetes.io/projected/8583d851-3b5b-49e9-9cc3-a1495455585b-kube-api-access-ncx9l\") pod \"8583d851-3b5b-49e9-9cc3-a1495455585b\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") "
Sep 17 15:47:35.301749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.301613    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-bundle\") pod \"21a37064-3cb5-4bc2-9aad-dc261c983967\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") "
Sep 17 15:47:35.301749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.301669    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-bundle\") pod \"8583d851-3b5b-49e9-9cc3-a1495455585b\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") "
Sep 17 15:47:35.301749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.301702    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-util\") pod \"eba480c7-9c34-4804-a7a8-37491fee90ea\" (UID: \"eba480c7-9c34-4804-a7a8-37491fee90ea\") "
Sep 17 15:47:35.301749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.301746    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-util\") pod \"21a37064-3cb5-4bc2-9aad-dc261c983967\" (UID: \"21a37064-3cb5-4bc2-9aad-dc261c983967\") "
Sep 17 15:47:35.302049 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.301797    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-util\") pod \"8583d851-3b5b-49e9-9cc3-a1495455585b\" (UID: \"8583d851-3b5b-49e9-9cc3-a1495455585b\") "
Sep 17 15:47:35.302459 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.302287    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-bundle" (OuterVolumeSpecName: "bundle") pod "21a37064-3cb5-4bc2-9aad-dc261c983967" (UID: "21a37064-3cb5-4bc2-9aad-dc261c983967"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.302459 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.302414    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-bundle" (OuterVolumeSpecName: "bundle") pod "eba480c7-9c34-4804-a7a8-37491fee90ea" (UID: "eba480c7-9c34-4804-a7a8-37491fee90ea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.302622 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.302571    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-bundle" (OuterVolumeSpecName: "bundle") pod "8583d851-3b5b-49e9-9cc3-a1495455585b" (UID: "8583d851-3b5b-49e9-9cc3-a1495455585b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.304106 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.304077    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba480c7-9c34-4804-a7a8-37491fee90ea-kube-api-access-zkdn5" (OuterVolumeSpecName: "kube-api-access-zkdn5") pod "eba480c7-9c34-4804-a7a8-37491fee90ea" (UID: "eba480c7-9c34-4804-a7a8-37491fee90ea"). InnerVolumeSpecName "kube-api-access-zkdn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:47:35.304183 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.304114    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a37064-3cb5-4bc2-9aad-dc261c983967-kube-api-access-6fq8b" (OuterVolumeSpecName: "kube-api-access-6fq8b") pod "21a37064-3cb5-4bc2-9aad-dc261c983967" (UID: "21a37064-3cb5-4bc2-9aad-dc261c983967"). InnerVolumeSpecName "kube-api-access-6fq8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:47:35.304526 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.304510    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8583d851-3b5b-49e9-9cc3-a1495455585b-kube-api-access-ncx9l" (OuterVolumeSpecName: "kube-api-access-ncx9l") pod "8583d851-3b5b-49e9-9cc3-a1495455585b" (UID: "8583d851-3b5b-49e9-9cc3-a1495455585b"). InnerVolumeSpecName "kube-api-access-ncx9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:47:35.307652 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.307631    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-util" (OuterVolumeSpecName: "util") pod "8583d851-3b5b-49e9-9cc3-a1495455585b" (UID: "8583d851-3b5b-49e9-9cc3-a1495455585b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.308270 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.308252    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-util" (OuterVolumeSpecName: "util") pod "21a37064-3cb5-4bc2-9aad-dc261c983967" (UID: "21a37064-3cb5-4bc2-9aad-dc261c983967"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.308350 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.308333    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-util" (OuterVolumeSpecName: "util") pod "eba480c7-9c34-4804-a7a8-37491fee90ea" (UID: "eba480c7-9c34-4804-a7a8-37491fee90ea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.403256 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.403236    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-util\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.403256 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.403256    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-bundle\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.403369 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.403265    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fq8b\" (UniqueName: \"kubernetes.io/projected/21a37064-3cb5-4bc2-9aad-dc261c983967-kube-api-access-6fq8b\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.403369 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.403274    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkdn5\" (UniqueName: \"kubernetes.io/projected/eba480c7-9c34-4804-a7a8-37491fee90ea-kube-api-access-zkdn5\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.403369 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.403283    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncx9l\" (UniqueName: \"kubernetes.io/projected/8583d851-3b5b-49e9-9cc3-a1495455585b-kube-api-access-ncx9l\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.403369 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.403292    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-bundle\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.403369 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.403300    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8583d851-3b5b-49e9-9cc3-a1495455585b-bundle\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.403369 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.403306    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eba480c7-9c34-4804-a7a8-37491fee90ea-util\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.403369 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:35.403313    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21a37064-3cb5-4bc2-9aad-dc261c983967-util\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:47:36.015692 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:36.015654    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst" event={"ID":"21a37064-3cb5-4bc2-9aad-dc261c983967","Type":"ContainerDied","Data":"1b489b18272bd2a72547ba5bf8e99636ac30fdbc5416cc42c7c9c86ecb9dc085"}
Sep 17 15:47:36.016171 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:36.015714    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b489b18272bd2a72547ba5bf8e99636ac30fdbc5416cc42c7c9c86ecb9dc085"
Sep 17 15:47:36.016171 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:36.015663    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68lmst"
Sep 17 15:47:36.017915 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:36.017892    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6"
Sep 17 15:47:36.018026 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:36.017928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847argbk6" event={"ID":"8583d851-3b5b-49e9-9cc3-a1495455585b","Type":"ContainerDied","Data":"959d94912d9c910ef9e53e46db1f949acc21e8b3d9d982db66c49c49f9967f4b"}
Sep 17 15:47:36.018026 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:36.017973    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="959d94912d9c910ef9e53e46db1f949acc21e8b3d9d982db66c49c49f9967f4b"
Sep 17 15:47:36.020560 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:36.020538    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5"
Sep 17 15:47:36.020699 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:36.020535    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbhcm5" event={"ID":"eba480c7-9c34-4804-a7a8-37491fee90ea","Type":"ContainerDied","Data":"1e5291fc5c2ddc4377fbbd2a40b17e8ae92368b4da0cf9571fe93d3d6a2f7d88"}
Sep 17 15:47:36.020699 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:36.020666    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e5291fc5c2ddc4377fbbd2a40b17e8ae92368b4da0cf9571fe93d3d6a2f7d88"
Sep 17 15:47:38.953675 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:47:38.953647    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65f7746b69-pcjww"
Sep 17 15:48:33.944439 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944406    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2k2v8"]
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944788    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba480c7-9c34-4804-a7a8-37491fee90ea" containerName="extract"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944805    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba480c7-9c34-4804-a7a8-37491fee90ea" containerName="extract"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944819    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8583d851-3b5b-49e9-9cc3-a1495455585b" containerName="util"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944848    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8583d851-3b5b-49e9-9cc3-a1495455585b" containerName="util"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944856    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba480c7-9c34-4804-a7a8-37491fee90ea" containerName="pull"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944867    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba480c7-9c34-4804-a7a8-37491fee90ea" containerName="pull"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944881    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a37064-3cb5-4bc2-9aad-dc261c983967" containerName="pull"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944888    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a37064-3cb5-4bc2-9aad-dc261c983967" containerName="pull"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944901    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8583d851-3b5b-49e9-9cc3-a1495455585b" containerName="extract"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944909    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8583d851-3b5b-49e9-9cc3-a1495455585b" containerName="extract"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944922    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c38fff-a1a2-4f59-b1f0-83327ea3061c" containerName="util"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944930    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c38fff-a1a2-4f59-b1f0-83327ea3061c" containerName="util"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944942    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8583d851-3b5b-49e9-9cc3-a1495455585b" containerName="pull"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944950    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8583d851-3b5b-49e9-9cc3-a1495455585b" containerName="pull"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944964    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c38fff-a1a2-4f59-b1f0-83327ea3061c" containerName="extract"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944972    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c38fff-a1a2-4f59-b1f0-83327ea3061c" containerName="extract"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944988    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba480c7-9c34-4804-a7a8-37491fee90ea" containerName="util"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.944996    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba480c7-9c34-4804-a7a8-37491fee90ea" containerName="util"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.945007    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c38fff-a1a2-4f59-b1f0-83327ea3061c" containerName="pull"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.945017    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c38fff-a1a2-4f59-b1f0-83327ea3061c" containerName="pull"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.945032    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a37064-3cb5-4bc2-9aad-dc261c983967" containerName="util"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.945039    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a37064-3cb5-4bc2-9aad-dc261c983967" containerName="util"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.945048    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a37064-3cb5-4bc2-9aad-dc261c983967" containerName="extract"
Sep 17 15:48:33.945052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.945056    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a37064-3cb5-4bc2-9aad-dc261c983967" containerName="extract"
Sep 17 15:48:33.946162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.945133    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a37064-3cb5-4bc2-9aad-dc261c983967" containerName="extract"
Sep 17 15:48:33.946162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.945149    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8583d851-3b5b-49e9-9cc3-a1495455585b" containerName="extract"
Sep 17 15:48:33.946162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.945158    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="56c38fff-a1a2-4f59-b1f0-83327ea3061c" containerName="extract"
Sep 17 15:48:33.946162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.945166    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba480c7-9c34-4804-a7a8-37491fee90ea" containerName="extract"
Sep 17 15:48:33.948092 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.948072    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8"
Sep 17 15:48:33.950911 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.950890    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rz4td\""
Sep 17 15:48:33.951005 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.950929    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 15:48:33.951979 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.951959    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 15:48:33.957417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.957400    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2k2v8"]
Sep 17 15:48:33.967777 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.967757    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-fc6x5"]
Sep 17 15:48:33.970648 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.970630    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fc6x5"
Sep 17 15:48:33.983181 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:33.983162    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fc6x5"]
Sep 17 15:48:34.027406 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:34.027387    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwc94\" (UniqueName: \"kubernetes.io/projected/83aa4bea-66c7-4004-a3b3-65b0e6765aa0-kube-api-access-lwc94\") pod \"authorino-7bcfd8f9cd-2k2v8\" (UID: \"83aa4bea-66c7-4004-a3b3-65b0e6765aa0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8"
Sep 17 15:48:34.128600 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:34.128574    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwc94\" (UniqueName: \"kubernetes.io/projected/83aa4bea-66c7-4004-a3b3-65b0e6765aa0-kube-api-access-lwc94\") pod \"authorino-7bcfd8f9cd-2k2v8\" (UID: \"83aa4bea-66c7-4004-a3b3-65b0e6765aa0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8"
Sep 17 15:48:34.128701 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:34.128617    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kllsx\" (UniqueName: \"kubernetes.io/projected/0102304d-52d4-497f-bf07-3a29955e9db5-kube-api-access-kllsx\") pod \"authorino-59c974bd48-fc6x5\" (UID: \"0102304d-52d4-497f-bf07-3a29955e9db5\") " pod="kuadrant-system/authorino-59c974bd48-fc6x5"
Sep 17 15:48:34.138631 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:34.138615    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwc94\" (UniqueName: \"kubernetes.io/projected/83aa4bea-66c7-4004-a3b3-65b0e6765aa0-kube-api-access-lwc94\") pod \"authorino-7bcfd8f9cd-2k2v8\" (UID: \"83aa4bea-66c7-4004-a3b3-65b0e6765aa0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8"
Sep 17 15:48:34.229868 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:34.229790    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kllsx\" (UniqueName: \"kubernetes.io/projected/0102304d-52d4-497f-bf07-3a29955e9db5-kube-api-access-kllsx\") pod \"authorino-59c974bd48-fc6x5\" (UID: \"0102304d-52d4-497f-bf07-3a29955e9db5\") " pod="kuadrant-system/authorino-59c974bd48-fc6x5"
Sep 17 15:48:34.240343 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:34.240325    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kllsx\" (UniqueName: \"kubernetes.io/projected/0102304d-52d4-497f-bf07-3a29955e9db5-kube-api-access-kllsx\") pod \"authorino-59c974bd48-fc6x5\" (UID: \"0102304d-52d4-497f-bf07-3a29955e9db5\") " pod="kuadrant-system/authorino-59c974bd48-fc6x5"
Sep 17 15:48:34.258200 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:34.258172    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8"
Sep 17 15:48:34.279159 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:34.279138    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fc6x5"
Sep 17 15:48:34.530895 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:34.530865    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2k2v8"]
Sep 17 15:48:34.532256 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:48:34.532232    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83aa4bea_66c7_4004_a3b3_65b0e6765aa0.slice/crio-9513f49b1937649ee80a52dfd7496f36af0e607a7734b018dbc6188793904692 WatchSource:0}: Error finding container 9513f49b1937649ee80a52dfd7496f36af0e607a7734b018dbc6188793904692: Status 404 returned error can't find the container with id 9513f49b1937649ee80a52dfd7496f36af0e607a7734b018dbc6188793904692
Sep 17 15:48:34.558654 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:34.558633    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fc6x5"]
Sep 17 15:48:34.558753 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:48:34.558706    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0102304d_52d4_497f_bf07_3a29955e9db5.slice/crio-ee87cde5f258bb20911cd8e78e2dc82cfddd4e99650f16e6389d39e642642fd0 WatchSource:0}: Error finding container ee87cde5f258bb20911cd8e78e2dc82cfddd4e99650f16e6389d39e642642fd0: Status 404 returned error can't find the container with id ee87cde5f258bb20911cd8e78e2dc82cfddd4e99650f16e6389d39e642642fd0
Sep 17 15:48:35.292999 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:35.292960    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fc6x5" event={"ID":"0102304d-52d4-497f-bf07-3a29955e9db5","Type":"ContainerStarted","Data":"ee87cde5f258bb20911cd8e78e2dc82cfddd4e99650f16e6389d39e642642fd0"}
Sep 17 15:48:35.294078 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:35.294052    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8" event={"ID":"83aa4bea-66c7-4004-a3b3-65b0e6765aa0","Type":"ContainerStarted","Data":"9513f49b1937649ee80a52dfd7496f36af0e607a7734b018dbc6188793904692"}
Sep 17 15:48:37.306624 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:37.306584    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fc6x5" event={"ID":"0102304d-52d4-497f-bf07-3a29955e9db5","Type":"ContainerStarted","Data":"b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8"}
Sep 17 15:48:37.322141 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:37.322093    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-fc6x5" podStartSLOduration=2.169436963 podStartE2EDuration="4.322076821s" totalImagesPullingTime="2.152639858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:37.320855846 +0000 UTC m=+599.211716413" watchObservedRunningTime="2026-09-17 15:48:37.322076821 +0000 UTC m=+599.212937365"
Sep 17 15:48:37.350986 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:37.350955    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2k2v8"]
Sep 17 15:48:38.313253 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:38.313216    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8" event={"ID":"83aa4bea-66c7-4004-a3b3-65b0e6765aa0","Type":"ContainerStarted","Data":"4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82"}
Sep 17 15:48:38.313253 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:38.313221    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8" podUID="83aa4bea-66c7-4004-a3b3-65b0e6765aa0" containerName="authorino" containerID="cri-o://4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82" gracePeriod=30
Sep 17 15:48:38.332234 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:38.332191    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8" podStartSLOduration=2.119038861 podStartE2EDuration="5.332175224s" totalImagesPullingTime="3.213136363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:38.33008679 +0000 UTC m=+600.220947410" watchObservedRunningTime="2026-09-17 15:48:38.332175224 +0000 UTC m=+600.223035767"
Sep 17 15:48:38.551737 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:38.551711    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8"
Sep 17 15:48:38.639151 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:38.639122    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:48:38.644971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:38.644945    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:48:38.663287 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:38.663258    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwc94\" (UniqueName: \"kubernetes.io/projected/83aa4bea-66c7-4004-a3b3-65b0e6765aa0-kube-api-access-lwc94\") pod \"83aa4bea-66c7-4004-a3b3-65b0e6765aa0\" (UID: \"83aa4bea-66c7-4004-a3b3-65b0e6765aa0\") "
Sep 17 15:48:38.665331 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:38.665309    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83aa4bea-66c7-4004-a3b3-65b0e6765aa0-kube-api-access-lwc94" (OuterVolumeSpecName: "kube-api-access-lwc94") pod "83aa4bea-66c7-4004-a3b3-65b0e6765aa0" (UID: "83aa4bea-66c7-4004-a3b3-65b0e6765aa0"). InnerVolumeSpecName "kube-api-access-lwc94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:48:38.763796 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:38.763764    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwc94\" (UniqueName: \"kubernetes.io/projected/83aa4bea-66c7-4004-a3b3-65b0e6765aa0-kube-api-access-lwc94\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:48:39.319523 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:39.319487    2555 generic.go:356] "Generic (PLEG): container finished" podID="83aa4bea-66c7-4004-a3b3-65b0e6765aa0" containerID="4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82" exitCode=0
Sep 17 15:48:39.320030 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:39.319546    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8"
Sep 17 15:48:39.320030 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:39.319579    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8" event={"ID":"83aa4bea-66c7-4004-a3b3-65b0e6765aa0","Type":"ContainerDied","Data":"4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82"}
Sep 17 15:48:39.320030 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:39.319647    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-2k2v8" event={"ID":"83aa4bea-66c7-4004-a3b3-65b0e6765aa0","Type":"ContainerDied","Data":"9513f49b1937649ee80a52dfd7496f36af0e607a7734b018dbc6188793904692"}
Sep 17 15:48:39.320030 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:39.319675    2555 scope.go:117] "RemoveContainer" containerID="4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82"
Sep 17 15:48:39.327878 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:39.327859    2555 scope.go:117] "RemoveContainer" containerID="4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82"
Sep 17 15:48:39.328213 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:48:39.328194    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82\": container with ID starting with 4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82 not found: ID does not exist" containerID="4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82"
Sep 17 15:48:39.328266 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:39.328221    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82"} err="failed to get container status \"4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82\": rpc error: code = NotFound desc = could not find container \"4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82\": container with ID starting with 4a5d0b0852fda4c03482761f0c727513bbe432f317f710b5fee073fe2929af82 not found: ID does not exist"
Sep 17 15:48:39.337574 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:39.337546    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2k2v8"]
Sep 17 15:48:39.341214 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:39.341197    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2k2v8"]
Sep 17 15:48:40.694381 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:40.694346    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83aa4bea-66c7-4004-a3b3-65b0e6765aa0" path="/var/lib/kubelet/pods/83aa4bea-66c7-4004-a3b3-65b0e6765aa0/volumes"
Sep 17 15:48:56.350023 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.349979    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5j4ml"]
Sep 17 15:48:56.353716 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.350408    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83aa4bea-66c7-4004-a3b3-65b0e6765aa0" containerName="authorino"
Sep 17 15:48:56.353716 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.350429    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83aa4bea-66c7-4004-a3b3-65b0e6765aa0" containerName="authorino"
Sep 17 15:48:56.353716 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.350515    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="83aa4bea-66c7-4004-a3b3-65b0e6765aa0" containerName="authorino"
Sep 17 15:48:56.355019 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.354996    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-5j4ml"
Sep 17 15:48:56.357760 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.357734    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 15:48:56.362100 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.362074    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5j4ml"]
Sep 17 15:48:56.489304 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.489281    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a84d3db-8d6e-4554-914b-184e8777dea7-tls-cert\") pod \"authorino-6d9898bd99-5j4ml\" (UID: \"2a84d3db-8d6e-4554-914b-184e8777dea7\") " pod="kuadrant-system/authorino-6d9898bd99-5j4ml"
Sep 17 15:48:56.489402 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.489311    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbnd\" (UniqueName: \"kubernetes.io/projected/2a84d3db-8d6e-4554-914b-184e8777dea7-kube-api-access-jdbnd\") pod \"authorino-6d9898bd99-5j4ml\" (UID: \"2a84d3db-8d6e-4554-914b-184e8777dea7\") " pod="kuadrant-system/authorino-6d9898bd99-5j4ml"
Sep 17 15:48:56.589726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.589700    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a84d3db-8d6e-4554-914b-184e8777dea7-tls-cert\") pod \"authorino-6d9898bd99-5j4ml\" (UID: \"2a84d3db-8d6e-4554-914b-184e8777dea7\") " pod="kuadrant-system/authorino-6d9898bd99-5j4ml"
Sep 17 15:48:56.589906 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.589733    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbnd\" (UniqueName: \"kubernetes.io/projected/2a84d3db-8d6e-4554-914b-184e8777dea7-kube-api-access-jdbnd\") pod \"authorino-6d9898bd99-5j4ml\" (UID: \"2a84d3db-8d6e-4554-914b-184e8777dea7\") " pod="kuadrant-system/authorino-6d9898bd99-5j4ml"
Sep 17 15:48:56.592171 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.592152    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a84d3db-8d6e-4554-914b-184e8777dea7-tls-cert\") pod \"authorino-6d9898bd99-5j4ml\" (UID: \"2a84d3db-8d6e-4554-914b-184e8777dea7\") " pod="kuadrant-system/authorino-6d9898bd99-5j4ml"
Sep 17 15:48:56.599405 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.599385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbnd\" (UniqueName: \"kubernetes.io/projected/2a84d3db-8d6e-4554-914b-184e8777dea7-kube-api-access-jdbnd\") pod \"authorino-6d9898bd99-5j4ml\" (UID: \"2a84d3db-8d6e-4554-914b-184e8777dea7\") " pod="kuadrant-system/authorino-6d9898bd99-5j4ml"
Sep 17 15:48:56.665355 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.665309    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-5j4ml"
Sep 17 15:48:56.939322 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:56.939297    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5j4ml"]
Sep 17 15:48:56.940639 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:48:56.940610    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a84d3db_8d6e_4554_914b_184e8777dea7.slice/crio-6c2e02c8226598415965852fcabafde670d23fcd58d227160d2ad7824b86c19e WatchSource:0}: Error finding container 6c2e02c8226598415965852fcabafde670d23fcd58d227160d2ad7824b86c19e: Status 404 returned error can't find the container with id 6c2e02c8226598415965852fcabafde670d23fcd58d227160d2ad7824b86c19e
Sep 17 15:48:57.407512 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:57.407485    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-5j4ml" event={"ID":"2a84d3db-8d6e-4554-914b-184e8777dea7","Type":"ContainerStarted","Data":"6c2e02c8226598415965852fcabafde670d23fcd58d227160d2ad7824b86c19e"}
Sep 17 15:48:58.413660 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:58.413616    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-5j4ml" event={"ID":"2a84d3db-8d6e-4554-914b-184e8777dea7","Type":"ContainerStarted","Data":"ad6bdd93885591a94b66e34d6cb6a3b7953292ea43b52c08f65d01ab2c2084bb"}
Sep 17 15:48:58.432702 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:58.432652    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-5j4ml" podStartSLOduration=1.971774331 podStartE2EDuration="2.432636824s" totalImagesPullingTime="460.862493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:58.432298765 +0000 UTC m=+620.323159307" watchObservedRunningTime="2026-09-17 15:48:58.432636824 +0000 UTC m=+620.323497368"
Sep 17 15:48:58.465248 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:58.465224    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fc6x5"]
Sep 17 15:48:58.465397 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:58.465379    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-fc6x5" podUID="0102304d-52d4-497f-bf07-3a29955e9db5" containerName="authorino" containerID="cri-o://b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8" gracePeriod=30
Sep 17 15:48:58.702896 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:58.702874    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fc6x5"
Sep 17 15:48:58.802533 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:58.802506    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kllsx\" (UniqueName: \"kubernetes.io/projected/0102304d-52d4-497f-bf07-3a29955e9db5-kube-api-access-kllsx\") pod \"0102304d-52d4-497f-bf07-3a29955e9db5\" (UID: \"0102304d-52d4-497f-bf07-3a29955e9db5\") "
Sep 17 15:48:58.804471 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:58.804441    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0102304d-52d4-497f-bf07-3a29955e9db5-kube-api-access-kllsx" (OuterVolumeSpecName: "kube-api-access-kllsx") pod "0102304d-52d4-497f-bf07-3a29955e9db5" (UID: "0102304d-52d4-497f-bf07-3a29955e9db5"). InnerVolumeSpecName "kube-api-access-kllsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:48:58.904011 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:58.903975    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kllsx\" (UniqueName: \"kubernetes.io/projected/0102304d-52d4-497f-bf07-3a29955e9db5-kube-api-access-kllsx\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:48:59.419670 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:59.419629    2555 generic.go:356] "Generic (PLEG): container finished" podID="0102304d-52d4-497f-bf07-3a29955e9db5" containerID="b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8" exitCode=0
Sep 17 15:48:59.419670 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:59.419669    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fc6x5"
Sep 17 15:48:59.420229 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:59.419682    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fc6x5" event={"ID":"0102304d-52d4-497f-bf07-3a29955e9db5","Type":"ContainerDied","Data":"b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8"}
Sep 17 15:48:59.420229 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:59.419723    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fc6x5" event={"ID":"0102304d-52d4-497f-bf07-3a29955e9db5","Type":"ContainerDied","Data":"ee87cde5f258bb20911cd8e78e2dc82cfddd4e99650f16e6389d39e642642fd0"}
Sep 17 15:48:59.420229 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:59.419739    2555 scope.go:117] "RemoveContainer" containerID="b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8"
Sep 17 15:48:59.428275 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:59.428262    2555 scope.go:117] "RemoveContainer" containerID="b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8"
Sep 17 15:48:59.428561 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:48:59.428544    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8\": container with ID starting with b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8 not found: ID does not exist" containerID="b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8"
Sep 17 15:48:59.428620 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:59.428573    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8"} err="failed to get container status \"b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8\": rpc error: code = NotFound desc = could not find container \"b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8\": container with ID starting with b8f621f02e142dfbfb27003dad825aed2c4295538864cc09668227461cc246f8 not found: ID does not exist"
Sep 17 15:48:59.443939 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:59.443920    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fc6x5"]
Sep 17 15:48:59.446634 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:48:59.446615    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fc6x5"]
Sep 17 15:49:00.693797 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:49:00.693721    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0102304d-52d4-497f-bf07-3a29955e9db5" path="/var/lib/kubelet/pods/0102304d-52d4-497f-bf07-3a29955e9db5/volumes"
Sep 17 15:52:54.569545 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:54.569483    2555 generic.go:356] "Generic (PLEG): container finished" podID="7e9582a6-488e-486d-a7ab-20ad6fb18be2" containerID="2951c63bfdaa5b2ac6def4e023bb8798e52bb6d5de28ce411839ea3a38965f2d" exitCode=0
Sep 17 15:52:54.569916 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:54.569560    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt" event={"ID":"7e9582a6-488e-486d-a7ab-20ad6fb18be2","Type":"ContainerDied","Data":"2951c63bfdaa5b2ac6def4e023bb8798e52bb6d5de28ce411839ea3a38965f2d"}
Sep 17 15:52:54.569961 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:54.569943    2555 scope.go:117] "RemoveContainer" containerID="2951c63bfdaa5b2ac6def4e023bb8798e52bb6d5de28ce411839ea3a38965f2d"
Sep 17 15:52:54.570990 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:54.570952    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 15:52:54.571098 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:54.571028    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 15:52:54.571098 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:54.571059    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 15:52:54.930426 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:54.930330    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:52:55.576562 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:55.576522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt" event={"ID":"7e9582a6-488e-486d-a7ab-20ad6fb18be2","Type":"ContainerStarted","Data":"c7fe7e453ae66f702d401e3c2961913056eeefd68332dd52c2fadbb79de427cf"}
Sep 17 15:52:56.069676 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:56.069576    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:52:56.070938 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:56.070907    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 17 15:52:56.071064 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:56.070974    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt" podUID="7e9582a6-488e-486d-a7ab-20ad6fb18be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 17 15:52:57.070040 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:57.070003    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 17 15:52:57.070417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:57.070059    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt" podUID="7e9582a6-488e-486d-a7ab-20ad6fb18be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 17 15:52:58.069988 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:58.069951    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 17 15:52:58.070159 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:58.070005    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt" podUID="7e9582a6-488e-486d-a7ab-20ad6fb18be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 17 15:52:59.069344 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:59.069305    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 17 15:52:59.069530 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:52:59.069363    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt" podUID="7e9582a6-488e-486d-a7ab-20ad6fb18be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 17 15:53:00.069403 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:00.069368    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 17 15:53:00.069815 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:00.069423    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt" podUID="7e9582a6-488e-486d-a7ab-20ad6fb18be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 17 15:53:01.073240 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:01.073210    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:53:01.073603 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:01.073505    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:53:01.074122 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:01.074103    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-crjlt"
Sep 17 15:53:03.901366 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:03.901331    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"]
Sep 17 15:53:03.901748 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:03.901606    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0102304d-52d4-497f-bf07-3a29955e9db5" containerName="authorino"
Sep 17 15:53:03.901748 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:03.901618    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0102304d-52d4-497f-bf07-3a29955e9db5" containerName="authorino"
Sep 17 15:53:03.901748 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:03.901680    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0102304d-52d4-497f-bf07-3a29955e9db5" containerName="authorino"
Sep 17 15:53:03.903655 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:03.903638    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:03.917414 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:03.917397    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-wkvd8\""
Sep 17 15:53:03.917714 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:03.917698    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 15:53:03.917862 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:03.917847    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 15:53:03.917953 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:03.917937    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 15:53:03.940246 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:03.940215    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"]
Sep 17 15:53:04.002895 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.002872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.002993 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.002899    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.002993 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.002923    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da724cbd-503c-483c-95df-f40e76145dac-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.002993 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.002960    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp78r\" (UniqueName: \"kubernetes.io/projected/da724cbd-503c-483c-95df-f40e76145dac-kube-api-access-vp78r\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.003129 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.003000    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.003129 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.003023    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.103988 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.103960    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp78r\" (UniqueName: \"kubernetes.io/projected/da724cbd-503c-483c-95df-f40e76145dac-kube-api-access-vp78r\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.104104 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.104001    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.104104 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.104025    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.104104 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.104048    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.104271 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.104158    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.104271 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.104201    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da724cbd-503c-483c-95df-f40e76145dac-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.104376 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.104343    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.104437 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.104389    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.104494 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.104447    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.106446 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.106428    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.106673 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.106653    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da724cbd-503c-483c-95df-f40e76145dac-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.117046 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.117024    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp78r\" (UniqueName: \"kubernetes.io/projected/da724cbd-503c-483c-95df-f40e76145dac-kube-api-access-vp78r\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-d448z\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.213213 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.213162    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:04.283860 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.283813    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"]
Sep 17 15:53:04.287530 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.287506    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:53:04.293634 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.293613    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-7wj8f\""
Sep 17 15:53:04.301908 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.301890    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"]
Sep 17 15:53:04.406874 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.406846    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpqw\" (UniqueName: \"kubernetes.io/projected/4565fc1c-bb77-43b1-89ec-8a35cf46d074-kube-api-access-jmpqw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc\" (UID: \"4565fc1c-bb77-43b1-89ec-8a35cf46d074\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:53:04.407023 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.406880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4565fc1c-bb77-43b1-89ec-8a35cf46d074-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc\" (UID: \"4565fc1c-bb77-43b1-89ec-8a35cf46d074\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:53:04.504346 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.504320    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"]
Sep 17 15:53:04.505437 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:53:04.505418    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda724cbd_503c_483c_95df_f40e76145dac.slice/crio-80d5a024e5b29fbe34266af149e77fa71a26b4b9694bb71e08c46865bb4582b6 WatchSource:0}: Error finding container 80d5a024e5b29fbe34266af149e77fa71a26b4b9694bb71e08c46865bb4582b6: Status 404 returned error can't find the container with id 80d5a024e5b29fbe34266af149e77fa71a26b4b9694bb71e08c46865bb4582b6
Sep 17 15:53:04.507304 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.507276    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:53:04.507385 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.507355    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpqw\" (UniqueName: \"kubernetes.io/projected/4565fc1c-bb77-43b1-89ec-8a35cf46d074-kube-api-access-jmpqw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc\" (UID: \"4565fc1c-bb77-43b1-89ec-8a35cf46d074\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:53:04.507445 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.507382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4565fc1c-bb77-43b1-89ec-8a35cf46d074-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc\" (UID: \"4565fc1c-bb77-43b1-89ec-8a35cf46d074\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:53:04.509846 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.509806    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4565fc1c-bb77-43b1-89ec-8a35cf46d074-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc\" (UID: \"4565fc1c-bb77-43b1-89ec-8a35cf46d074\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:53:04.536788 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.536768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpqw\" (UniqueName: \"kubernetes.io/projected/4565fc1c-bb77-43b1-89ec-8a35cf46d074-kube-api-access-jmpqw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc\" (UID: \"4565fc1c-bb77-43b1-89ec-8a35cf46d074\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:53:04.597691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.597674    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:53:04.619143 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.619120    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z" event={"ID":"da724cbd-503c-483c-95df-f40e76145dac","Type":"ContainerStarted","Data":"80d5a024e5b29fbe34266af149e77fa71a26b4b9694bb71e08c46865bb4582b6"}
Sep 17 15:53:04.886137 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:04.886113    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"]
Sep 17 15:53:04.909010 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:53:04.908980    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4565fc1c_bb77_43b1_89ec_8a35cf46d074.slice/crio-e1bd6bf2cc34ad82acc85973d157f98e07365dff94d9764a1a9aa402d9300149 WatchSource:0}: Error finding container e1bd6bf2cc34ad82acc85973d157f98e07365dff94d9764a1a9aa402d9300149: Status 404 returned error can't find the container with id e1bd6bf2cc34ad82acc85973d157f98e07365dff94d9764a1a9aa402d9300149
Sep 17 15:53:05.626456 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:05.626418    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" event={"ID":"4565fc1c-bb77-43b1-89ec-8a35cf46d074","Type":"ContainerStarted","Data":"e1bd6bf2cc34ad82acc85973d157f98e07365dff94d9764a1a9aa402d9300149"}
Sep 17 15:53:06.633756 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:06.633716    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z" event={"ID":"da724cbd-503c-483c-95df-f40e76145dac","Type":"ContainerStarted","Data":"204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff"}
Sep 17 15:53:07.213561 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:07.213524    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:07.640098 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:07.640064    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" event={"ID":"4565fc1c-bb77-43b1-89ec-8a35cf46d074","Type":"ContainerStarted","Data":"78ba2c9baa9b055fcc903f988fa584fad6c9bb5e6e667f1148cc8575352c1935"}
Sep 17 15:53:07.640464 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:07.640160    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:53:07.688006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:07.687960    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z" podStartSLOduration=3.146347146 podStartE2EDuration="4.687946643s" totalImagesPullingTime="1.541599497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:06.671872566 +0000 UTC m=+868.562733113" watchObservedRunningTime="2026-09-17 15:53:07.687946643 +0000 UTC m=+869.578807240"
Sep 17 15:53:07.688774 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:07.688744    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" podStartSLOduration=1.209659104 podStartE2EDuration="3.688738011s" totalImagesPullingTime="2.479078907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:07.687300972 +0000 UTC m=+869.578161516" watchObservedRunningTime="2026-09-17 15:53:07.688738011 +0000 UTC m=+869.579598554"
Sep 17 15:53:14.213764 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:14.213725    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:14.228808 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:14.228778    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:14.685496 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:14.685469    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:38.647588 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:38.647502    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:53:38.687392 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:38.687363    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:53:38.690997 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:38.690975    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:53:45.795130 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:45.795092    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"]
Sep 17 15:53:45.804232 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:53:45.804194    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 15:53:45.804232 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:53:45.804232 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:53:45.804232 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff" cmd=["/bin/sleep","15"]
Sep 17 15:53:45.804473 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:45.804255    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z" message="" err=<
Sep 17 15:53:45.804473 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:53:45.804473 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:53:45.804473 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 15:53:45.804473 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:53:45.804273    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 15:53:45.804473 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:53:45.804473 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:53:45.804473 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z" podUID="da724cbd-503c-483c-95df-f40e76145dac" containerName="main" containerID="cri-o://204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff"
Sep 17 15:53:45.804473 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:45.804323    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z" podUID="da724cbd-503c-483c-95df-f40e76145dac" containerName="main" containerID="cri-o://204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff" gracePeriod=60
Sep 17 15:53:45.805978 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:45.805958    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"]
Sep 17 15:53:45.808006 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:45.807979    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:53:46.143244 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.143224    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:46.203282 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.203259    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp78r\" (UniqueName: \"kubernetes.io/projected/da724cbd-503c-483c-95df-f40e76145dac-kube-api-access-vp78r\") pod \"da724cbd-503c-483c-95df-f40e76145dac\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") "
Sep 17 15:53:46.203403 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.203304    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-model-cache\") pod \"da724cbd-503c-483c-95df-f40e76145dac\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") "
Sep 17 15:53:46.203403 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.203326    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da724cbd-503c-483c-95df-f40e76145dac-tls-certs\") pod \"da724cbd-503c-483c-95df-f40e76145dac\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") "
Sep 17 15:53:46.203403 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.203349    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-home\") pod \"da724cbd-503c-483c-95df-f40e76145dac\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") "
Sep 17 15:53:46.203403 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.203393    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-dshm\") pod \"da724cbd-503c-483c-95df-f40e76145dac\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") "
Sep 17 15:53:46.203618 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.203469    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-tmp-dir\") pod \"da724cbd-503c-483c-95df-f40e76145dac\" (UID: \"da724cbd-503c-483c-95df-f40e76145dac\") "
Sep 17 15:53:46.203618 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.203517    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-model-cache" (OuterVolumeSpecName: "model-cache") pod "da724cbd-503c-483c-95df-f40e76145dac" (UID: "da724cbd-503c-483c-95df-f40e76145dac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:46.203718 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.203678    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:53:46.203718 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.203679    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-home" (OuterVolumeSpecName: "home") pod "da724cbd-503c-483c-95df-f40e76145dac" (UID: "da724cbd-503c-483c-95df-f40e76145dac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:46.203718 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.203700    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da724cbd-503c-483c-95df-f40e76145dac" (UID: "da724cbd-503c-483c-95df-f40e76145dac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:46.205649 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.205619    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-dshm" (OuterVolumeSpecName: "dshm") pod "da724cbd-503c-483c-95df-f40e76145dac" (UID: "da724cbd-503c-483c-95df-f40e76145dac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:46.205757 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.205703    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da724cbd-503c-483c-95df-f40e76145dac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da724cbd-503c-483c-95df-f40e76145dac" (UID: "da724cbd-503c-483c-95df-f40e76145dac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:53:46.205757 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.205728    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da724cbd-503c-483c-95df-f40e76145dac-kube-api-access-vp78r" (OuterVolumeSpecName: "kube-api-access-vp78r") pod "da724cbd-503c-483c-95df-f40e76145dac" (UID: "da724cbd-503c-483c-95df-f40e76145dac"). InnerVolumeSpecName "kube-api-access-vp78r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:53:46.304071 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.304045    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:53:46.304071 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.304067    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp78r\" (UniqueName: \"kubernetes.io/projected/da724cbd-503c-483c-95df-f40e76145dac-kube-api-access-vp78r\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:53:46.304202 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.304080    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da724cbd-503c-483c-95df-f40e76145dac-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:53:46.304202 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.304094    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:53:46.304202 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.304104    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da724cbd-503c-483c-95df-f40e76145dac-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:53:46.824224 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.824188    2555 generic.go:356] "Generic (PLEG): container finished" podID="da724cbd-503c-483c-95df-f40e76145dac" containerID="204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff" exitCode=0
Sep 17 15:53:46.824684 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.824235    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"
Sep 17 15:53:46.824684 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.824279    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z" event={"ID":"da724cbd-503c-483c-95df-f40e76145dac","Type":"ContainerDied","Data":"204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff"}
Sep 17 15:53:46.824684 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.824343    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z" event={"ID":"da724cbd-503c-483c-95df-f40e76145dac","Type":"ContainerDied","Data":"80d5a024e5b29fbe34266af149e77fa71a26b4b9694bb71e08c46865bb4582b6"}
Sep 17 15:53:46.824684 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.824378    2555 scope.go:117] "RemoveContainer" containerID="204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff"
Sep 17 15:53:46.832788 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.832767    2555 scope.go:117] "RemoveContainer" containerID="204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff"
Sep 17 15:53:46.833081 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:53:46.833063    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff\": container with ID starting with 204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff not found: ID does not exist" containerID="204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff"
Sep 17 15:53:46.833134 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.833091    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff"} err="failed to get container status \"204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff\": rpc error: code = NotFound desc = could not find container \"204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff\": container with ID starting with 204528795de88142e42693b80858a69a9535111563ebe558e12fac0d0f4332ff not found: ID does not exist"
Sep 17 15:53:46.859450 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.859420    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"]
Sep 17 15:53:46.872317 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:46.872289    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-d448z"]
Sep 17 15:53:48.694477 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:48.694440    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da724cbd-503c-483c-95df-f40e76145dac" path="/var/lib/kubelet/pods/da724cbd-503c-483c-95df-f40e76145dac/volumes"
Sep 17 15:53:55.808521 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:53:55.808484    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:54:00.823851 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:00.823793    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerName="main" containerID="cri-o://78ba2c9baa9b055fcc903f988fa584fad6c9bb5e6e667f1148cc8575352c1935" gracePeriod=45
Sep 17 15:54:05.807441 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:05.807402    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:54:15.807857 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:15.807804    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:54:25.808417 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:25.808381    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:54:31.036278 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:31.036244    2555 generic.go:356] "Generic (PLEG): container finished" podID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerID="78ba2c9baa9b055fcc903f988fa584fad6c9bb5e6e667f1148cc8575352c1935" exitCode=0
Sep 17 15:54:31.036612 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:31.036310    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" event={"ID":"4565fc1c-bb77-43b1-89ec-8a35cf46d074","Type":"ContainerDied","Data":"78ba2c9baa9b055fcc903f988fa584fad6c9bb5e6e667f1148cc8575352c1935"}
Sep 17 15:54:31.065286 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:31.065264    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:54:31.223392 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:31.223309    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4565fc1c-bb77-43b1-89ec-8a35cf46d074-tls-certs\") pod \"4565fc1c-bb77-43b1-89ec-8a35cf46d074\" (UID: \"4565fc1c-bb77-43b1-89ec-8a35cf46d074\") "
Sep 17 15:54:31.223392 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:31.223372    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmpqw\" (UniqueName: \"kubernetes.io/projected/4565fc1c-bb77-43b1-89ec-8a35cf46d074-kube-api-access-jmpqw\") pod \"4565fc1c-bb77-43b1-89ec-8a35cf46d074\" (UID: \"4565fc1c-bb77-43b1-89ec-8a35cf46d074\") "
Sep 17 15:54:31.225397 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:31.225364    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4565fc1c-bb77-43b1-89ec-8a35cf46d074-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4565fc1c-bb77-43b1-89ec-8a35cf46d074" (UID: "4565fc1c-bb77-43b1-89ec-8a35cf46d074"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:31.225397 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:31.225377    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4565fc1c-bb77-43b1-89ec-8a35cf46d074-kube-api-access-jmpqw" (OuterVolumeSpecName: "kube-api-access-jmpqw") pod "4565fc1c-bb77-43b1-89ec-8a35cf46d074" (UID: "4565fc1c-bb77-43b1-89ec-8a35cf46d074"). InnerVolumeSpecName "kube-api-access-jmpqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:31.324000 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:31.323977    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpqw\" (UniqueName: \"kubernetes.io/projected/4565fc1c-bb77-43b1-89ec-8a35cf46d074-kube-api-access-jmpqw\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:54:31.324000 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:31.323998    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4565fc1c-bb77-43b1-89ec-8a35cf46d074-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:54:32.042744 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:32.042698    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc" event={"ID":"4565fc1c-bb77-43b1-89ec-8a35cf46d074","Type":"ContainerDied","Data":"e1bd6bf2cc34ad82acc85973d157f98e07365dff94d9764a1a9aa402d9300149"}
Sep 17 15:54:32.043262 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:32.042759    2555 scope.go:117] "RemoveContainer" containerID="78ba2c9baa9b055fcc903f988fa584fad6c9bb5e6e667f1148cc8575352c1935"
Sep 17 15:54:32.043262 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:32.042715    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"
Sep 17 15:54:32.067661 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:32.067635    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"]
Sep 17 15:54:32.076253 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:32.076227    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5shvpc"]
Sep 17 15:54:32.694705 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:54:32.694668    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" path="/var/lib/kubelet/pods/4565fc1c-bb77-43b1-89ec-8a35cf46d074/volumes"
Sep 17 15:55:29.591305 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.591269    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"]
Sep 17 15:55:29.591749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.591550    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerName="main"
Sep 17 15:55:29.591749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.591561    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerName="main"
Sep 17 15:55:29.591749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.591568    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da724cbd-503c-483c-95df-f40e76145dac" containerName="main"
Sep 17 15:55:29.591749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.591574    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="da724cbd-503c-483c-95df-f40e76145dac" containerName="main"
Sep 17 15:55:29.591749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.591628    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="da724cbd-503c-483c-95df-f40e76145dac" containerName="main"
Sep 17 15:55:29.591749 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.591639    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4565fc1c-bb77-43b1-89ec-8a35cf46d074" containerName="main"
Sep 17 15:55:29.593672 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.593655    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.597649 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.597612    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 15:55:29.597649 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.597639    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-jppsv\""
Sep 17 15:55:29.597872 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.597641    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 15:55:29.597872 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.597791    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 15:55:29.605977 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.605954    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"]
Sep 17 15:55:29.629429 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.629407    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.629572 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.629440    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-home\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.629572 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.629480    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.629572 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.629552    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.629738 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.629618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.629738 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.629647    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbrv\" (UniqueName: \"kubernetes.io/projected/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-kube-api-access-grbrv\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.730815 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.730787    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.730976 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.730861    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.730976 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.730893    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbrv\" (UniqueName: \"kubernetes.io/projected/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-kube-api-access-grbrv\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.730976 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.730921    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.730976 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.730957    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-home\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.731205 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.731004    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.731260 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.731221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.731332 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.731308    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-home\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.731386 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.731354    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.733157 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.733132    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.733399 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.733381    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.740840 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.740803    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbrv\" (UniqueName: \"kubernetes.io/projected/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-kube-api-access-grbrv\") pod \"fc-concurrency-test-kserve-b5d44467c-dlbtr\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:29.903874 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:29.903806    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:30.179673 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:30.179644    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"]
Sep 17 15:55:30.184221 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:55:30.184100    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7c22b5_fdfd_4805_ada2_4faf06c1d1ad.slice/crio-b2b01b7ca51ab7c745c645456d26c882dde4c2c014f1d88476d9fa64089a8b6c WatchSource:0}: Error finding container b2b01b7ca51ab7c745c645456d26c882dde4c2c014f1d88476d9fa64089a8b6c: Status 404 returned error can't find the container with id b2b01b7ca51ab7c745c645456d26c882dde4c2c014f1d88476d9fa64089a8b6c
Sep 17 15:55:30.319797 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:30.319765    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr" event={"ID":"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad","Type":"ContainerStarted","Data":"5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f"}
Sep 17 15:55:30.319971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:30.319814    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr" event={"ID":"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad","Type":"ContainerStarted","Data":"b2b01b7ca51ab7c745c645456d26c882dde4c2c014f1d88476d9fa64089a8b6c"}
Sep 17 15:55:30.340450 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:30.340402    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr" podStartSLOduration=1.340387782 podStartE2EDuration="1.340387782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:30.339534932 +0000 UTC m=+1012.230395475" watchObservedRunningTime="2026-09-17 15:55:30.340387782 +0000 UTC m=+1012.231248326"
Sep 17 15:55:30.904309 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:30.904275    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:39.904998 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:39.904965    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:39.917124 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:39.917093    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:55:40.381727 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:55:40.381699    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:56:08.684713 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:08.684676    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"]
Sep 17 15:56:08.697692 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:56:08.697654    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 15:56:08.697692 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:56:08.697692 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:56:08.697692 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f" cmd=["/bin/sleep","15"]
Sep 17 15:56:08.698046 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:08.697706    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr" message="" err=<
Sep 17 15:56:08.698046 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:56:08.698046 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:56:08.698046 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 15:56:08.698046 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:56:08.697723    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 15:56:08.698046 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:56:08.698046 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:56:08.698046 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr" podUID="bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" containerName="main" containerID="cri-o://5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f"
Sep 17 15:56:08.698046 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:08.697758    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr" podUID="bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" containerName="main" containerID="cri-o://5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f" gracePeriod=60
Sep 17 15:56:09.429335 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.429312    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:56:09.506882 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.506850    2555 generic.go:356] "Generic (PLEG): container finished" podID="bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" containerID="5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f" exitCode=0
Sep 17 15:56:09.507008 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.506888    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"
Sep 17 15:56:09.507008 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.506925    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr" event={"ID":"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad","Type":"ContainerDied","Data":"5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f"}
Sep 17 15:56:09.507008 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.506979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr" event={"ID":"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad","Type":"ContainerDied","Data":"b2b01b7ca51ab7c745c645456d26c882dde4c2c014f1d88476d9fa64089a8b6c"}
Sep 17 15:56:09.507008 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.507008    2555 scope.go:117] "RemoveContainer" containerID="5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f"
Sep 17 15:56:09.515754 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.515739    2555 scope.go:117] "RemoveContainer" containerID="5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f"
Sep 17 15:56:09.516024 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:56:09.516007    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f\": container with ID starting with 5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f not found: ID does not exist" containerID="5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f"
Sep 17 15:56:09.516082 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.516032    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f"} err="failed to get container status \"5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f\": rpc error: code = NotFound desc = could not find container \"5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f\": container with ID starting with 5a6992417350b08ee472861124e2c50945ff3b8f3f31c1496d759cfecc2d115f not found: ID does not exist"
Sep 17 15:56:09.520292 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.520276    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-home\") pod \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") "
Sep 17 15:56:09.520387 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.520307    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tls-certs\") pod \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") "
Sep 17 15:56:09.520387 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.520329    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-dshm\") pod \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") "
Sep 17 15:56:09.520387 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.520350    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tmp-dir\") pod \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") "
Sep 17 15:56:09.520546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.520408    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-model-cache\") pod \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") "
Sep 17 15:56:09.520546 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.520425    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grbrv\" (UniqueName: \"kubernetes.io/projected/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-kube-api-access-grbrv\") pod \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\" (UID: \"bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad\") "
Sep 17 15:56:09.520651 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.520550    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-home" (OuterVolumeSpecName: "home") pod "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" (UID: "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:09.520712 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.520653    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" (UID: "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:09.520712 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.520661    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:56:09.520712 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.520671    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" (UID: "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:09.522585 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.522560    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" (UID: "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:09.522662 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.522552    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-kube-api-access-grbrv" (OuterVolumeSpecName: "kube-api-access-grbrv") pod "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" (UID: "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad"). InnerVolumeSpecName "kube-api-access-grbrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:09.522662 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.522609    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-dshm" (OuterVolumeSpecName: "dshm") pod "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" (UID: "bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:09.621260 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.621237    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:56:09.621260 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.621260    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grbrv\" (UniqueName: \"kubernetes.io/projected/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-kube-api-access-grbrv\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:56:09.621395 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.621271    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:56:09.621395 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.621279    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:56:09.621395 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.621288    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:56:09.836209 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.836183    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"]
Sep 17 15:56:09.840102 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:09.840078    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr"]
Sep 17 15:56:10.371926 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:10.371881    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dlbtr" podUID="bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": context deadline exceeded"
Sep 17 15:56:10.694316 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:10.694239    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" path="/var/lib/kubelet/pods/bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad/volumes"
Sep 17 15:56:54.213762 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.213733    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"]
Sep 17 15:56:54.214240 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.214051    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" containerName="main"
Sep 17 15:56:54.214240 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.214064    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" containerName="main"
Sep 17 15:56:54.214240 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.214123    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7c22b5-fdfd-4805-ada2-4faf06c1d1ad" containerName="main"
Sep 17 15:56:54.218092 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.218077    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.222053 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.222032    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 15:56:54.222053 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.222050    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 15:56:54.222246 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.222051    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-zvx8v\""
Sep 17 15:56:54.222246 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.222115    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 15:56:54.227788 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.227766    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"]
Sep 17 15:56:54.318338 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.318315    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f744659a-f38b-4316-8de3-7d717891a34c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.318432 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.318353    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.318432 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.318374    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl888\" (UniqueName: \"kubernetes.io/projected/f744659a-f38b-4316-8de3-7d717891a34c-kube-api-access-tl888\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.318508 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.318449    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.318508 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.318477    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.318508 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.318495    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.419551 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.419520    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.419551 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.419550    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.419715 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.419581    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f744659a-f38b-4316-8de3-7d717891a34c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.419715 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.419624    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.419715 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.419654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl888\" (UniqueName: \"kubernetes.io/projected/f744659a-f38b-4316-8de3-7d717891a34c-kube-api-access-tl888\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.419715 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.419702    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.420084 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.419987    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.420084 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.420016    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.420084 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.420070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.422337 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.422317    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.422712 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.422692    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f744659a-f38b-4316-8de3-7d717891a34c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.436001 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.435980    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl888\" (UniqueName: \"kubernetes.io/projected/f744659a-f38b-4316-8de3-7d717891a34c-kube-api-access-tl888\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.529572 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.529520    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:54.807306 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:54.807283    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"]
Sep 17 15:56:54.808788 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:56:54.808758    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf744659a_f38b_4316_8de3_7d717891a34c.slice/crio-ffae8a1a44fe33140c164dbbc1d1fe4789e08ac3ddb239822527d9361d5ad1ce WatchSource:0}: Error finding container ffae8a1a44fe33140c164dbbc1d1fe4789e08ac3ddb239822527d9361d5ad1ce: Status 404 returned error can't find the container with id ffae8a1a44fe33140c164dbbc1d1fe4789e08ac3ddb239822527d9361d5ad1ce
Sep 17 15:56:55.742585 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:55.742535    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp" event={"ID":"f744659a-f38b-4316-8de3-7d717891a34c","Type":"ContainerStarted","Data":"71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec"}
Sep 17 15:56:55.743132 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:55.742612    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp" event={"ID":"f744659a-f38b-4316-8de3-7d717891a34c","Type":"ContainerStarted","Data":"ffae8a1a44fe33140c164dbbc1d1fe4789e08ac3ddb239822527d9361d5ad1ce"}
Sep 17 15:56:55.765816 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:55.765762    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp" podStartSLOduration=1.765745522 podStartE2EDuration="1.765745522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:55.763632675 +0000 UTC m=+1097.654493234" watchObservedRunningTime="2026-09-17 15:56:55.765745522 +0000 UTC m=+1097.656606057"
Sep 17 15:56:56.530169 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:56.530131    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:56:59.016543 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.016509    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"]
Sep 17 15:56:59.019958 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.019938    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.025083 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.025040    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 15:56:59.025200 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.025162    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-swxrz\""
Sep 17 15:56:59.025591 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.025568    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 15:56:59.026015 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.025998    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 15:56:59.058890 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.057655    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"]
Sep 17 15:56:59.167502 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.167470    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.167703 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.167601    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d31984-95cc-4dbb-9468-e87d684e19ca-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.167703 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.167682    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.167855 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.167717    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rdw\" (UniqueName: \"kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-kube-api-access-82rdw\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.167855 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.167771    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.167974 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.167893    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.167974 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.167942    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.168078 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.168001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.168078 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.168048    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.268647 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.268556    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.268647 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.268598    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.268920 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.268898    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269033 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.269022    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269110 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.269101    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d31984-95cc-4dbb-9468-e87d684e19ca-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269187 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.269178    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269289 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.269273    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rdw\" (UniqueName: \"kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-kube-api-access-82rdw\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269289 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.268936    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269394 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.269279    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269394 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.269308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269394 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.269116    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269551 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.269416    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269746 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.269727    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d31984-95cc-4dbb-9468-e87d684e19ca-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.269923 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.269752    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.270790 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.270773    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.271539 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.271523    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.278359 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.278331    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.279052 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.279031    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rdw\" (UniqueName: \"kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-kube-api-access-82rdw\") pod \"router-gateway-1-openshift-default-6b46477c5b-4dnll\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.357278 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.357252    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:56:59.635928 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.635818    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"]
Sep 17 15:56:59.637108 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:56:59.637078    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d31984_95cc_4dbb_9468_e87d684e19ca.slice/crio-2d47de5544b11048e61aa09c4b376bceb54a5dc3ba6ef5c4c12b11a9d4f2d8fa WatchSource:0}: Error finding container 2d47de5544b11048e61aa09c4b376bceb54a5dc3ba6ef5c4c12b11a9d4f2d8fa: Status 404 returned error can't find the container with id 2d47de5544b11048e61aa09c4b376bceb54a5dc3ba6ef5c4c12b11a9d4f2d8fa
Sep 17 15:56:59.640569 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.640520    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 15:56:59.640667 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.640604    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 15:56:59.640667 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.640655    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 15:56:59.769587 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.769544    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll" event={"ID":"99d31984-95cc-4dbb-9468-e87d684e19ca","Type":"ContainerStarted","Data":"fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a"}
Sep 17 15:56:59.769789 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.769614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll" event={"ID":"99d31984-95cc-4dbb-9468-e87d684e19ca","Type":"ContainerStarted","Data":"2d47de5544b11048e61aa09c4b376bceb54a5dc3ba6ef5c4c12b11a9d4f2d8fa"}
Sep 17 15:56:59.795803 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:56:59.795728    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll" podStartSLOduration=1.795712427 podStartE2EDuration="1.795712427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:59.795045431 +0000 UTC m=+1101.685905974" watchObservedRunningTime="2026-09-17 15:56:59.795712427 +0000 UTC m=+1101.686572971"
Sep 17 15:57:00.358642 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:00.358605    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:57:00.363415 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:00.363390    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:57:00.776935 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:00.776818    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:57:00.777907 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:00.777882    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:57:04.530614 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:04.530575    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:57:04.543083 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:04.543058    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:57:04.807377 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:04.807306    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:57:18.261868 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.261819    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"]
Sep 17 15:57:18.264867 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.264850    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.268695 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.268666    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 15:57:18.268857 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.268669    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-g9wdz\""
Sep 17 15:57:18.279710 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.279679    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"]
Sep 17 15:57:18.406522 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.406485    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.406680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.406535    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.406680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.406570    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.406680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.406606    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0181d82d-82d2-4dd5-836c-6377670bd2c1-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.406680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.406666    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8xml\" (UniqueName: \"kubernetes.io/projected/0181d82d-82d2-4dd5-836c-6377670bd2c1-kube-api-access-h8xml\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.406887 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.406710    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.507256 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.507224    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.507460 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.507276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.507460 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.507308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.507460 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.507324    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.507460 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.507341    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0181d82d-82d2-4dd5-836c-6377670bd2c1-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.507460 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.507373    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8xml\" (UniqueName: \"kubernetes.io/projected/0181d82d-82d2-4dd5-836c-6377670bd2c1-kube-api-access-h8xml\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.507715 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.507682    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.507774 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.507752    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.507812 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.507780    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.509691 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.509669    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.509854 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.509836    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0181d82d-82d2-4dd5-836c-6377670bd2c1-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.517593 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.517534    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8xml\" (UniqueName: \"kubernetes.io/projected/0181d82d-82d2-4dd5-836c-6377670bd2c1-kube-api-access-h8xml\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.575539 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.575514    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:18.856087 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.856060    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"]
Sep 17 15:57:18.857725 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:57:18.857701    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0181d82d_82d2_4dd5_836c_6377670bd2c1.slice/crio-e2ea9a5550c3a47c7e07117cb68484234367332f18e98e653bcc98158de8412e WatchSource:0}: Error finding container e2ea9a5550c3a47c7e07117cb68484234367332f18e98e653bcc98158de8412e: Status 404 returned error can't find the container with id e2ea9a5550c3a47c7e07117cb68484234367332f18e98e653bcc98158de8412e
Sep 17 15:57:18.867324 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:18.867296    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2" event={"ID":"0181d82d-82d2-4dd5-836c-6377670bd2c1","Type":"ContainerStarted","Data":"e2ea9a5550c3a47c7e07117cb68484234367332f18e98e653bcc98158de8412e"}
Sep 17 15:57:19.873664 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:19.873620    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2" event={"ID":"0181d82d-82d2-4dd5-836c-6377670bd2c1","Type":"ContainerStarted","Data":"a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5"}
Sep 17 15:57:19.897063 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:19.897015    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2" podStartSLOduration=1.89699944 podStartE2EDuration="1.89699944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:19.895213007 +0000 UTC m=+1121.786073564" watchObservedRunningTime="2026-09-17 15:57:19.89699944 +0000 UTC m=+1121.787859984"
Sep 17 15:57:20.576326 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:20.576294    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:24.161623 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.161590    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"]
Sep 17 15:57:24.170999 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:57:24.170954    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 15:57:24.170999 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:57:24.170999 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:57:24.170999 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5" cmd=["/bin/sleep","15"]
Sep 17 15:57:24.171254 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.171018    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2" message="" err=<
Sep 17 15:57:24.171254 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:57:24.171254 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:57:24.171254 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 15:57:24.171254 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:57:24.171039    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 15:57:24.171254 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:57:24.171254 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:57:24.171254 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2" podUID="0181d82d-82d2-4dd5-836c-6377670bd2c1" containerName="main" containerID="cri-o://a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5"
Sep 17 15:57:24.171254 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.171089    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2" podUID="0181d82d-82d2-4dd5-836c-6377670bd2c1" containerName="main" containerID="cri-o://a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5" gracePeriod=60
Sep 17 15:57:24.426218 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.426135    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:24.563788 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.563754    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-dshm\") pod \"0181d82d-82d2-4dd5-836c-6377670bd2c1\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") "
Sep 17 15:57:24.563788 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.563788    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-model-cache\") pod \"0181d82d-82d2-4dd5-836c-6377670bd2c1\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") "
Sep 17 15:57:24.564077 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.563812    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8xml\" (UniqueName: \"kubernetes.io/projected/0181d82d-82d2-4dd5-836c-6377670bd2c1-kube-api-access-h8xml\") pod \"0181d82d-82d2-4dd5-836c-6377670bd2c1\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") "
Sep 17 15:57:24.564077 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.563872    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0181d82d-82d2-4dd5-836c-6377670bd2c1-tls-certs\") pod \"0181d82d-82d2-4dd5-836c-6377670bd2c1\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") "
Sep 17 15:57:24.564077 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.563892    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-tmp-dir\") pod \"0181d82d-82d2-4dd5-836c-6377670bd2c1\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") "
Sep 17 15:57:24.564077 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.563921    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-home\") pod \"0181d82d-82d2-4dd5-836c-6377670bd2c1\" (UID: \"0181d82d-82d2-4dd5-836c-6377670bd2c1\") "
Sep 17 15:57:24.564297 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.564186    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-home" (OuterVolumeSpecName: "home") pod "0181d82d-82d2-4dd5-836c-6377670bd2c1" (UID: "0181d82d-82d2-4dd5-836c-6377670bd2c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:24.564590 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.564563    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0181d82d-82d2-4dd5-836c-6377670bd2c1" (UID: "0181d82d-82d2-4dd5-836c-6377670bd2c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:24.564590 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.564579    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "0181d82d-82d2-4dd5-836c-6377670bd2c1" (UID: "0181d82d-82d2-4dd5-836c-6377670bd2c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:24.566324 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.566302    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-dshm" (OuterVolumeSpecName: "dshm") pod "0181d82d-82d2-4dd5-836c-6377670bd2c1" (UID: "0181d82d-82d2-4dd5-836c-6377670bd2c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:24.566443 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.566423    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0181d82d-82d2-4dd5-836c-6377670bd2c1-kube-api-access-h8xml" (OuterVolumeSpecName: "kube-api-access-h8xml") pod "0181d82d-82d2-4dd5-836c-6377670bd2c1" (UID: "0181d82d-82d2-4dd5-836c-6377670bd2c1"). InnerVolumeSpecName "kube-api-access-h8xml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:24.566993 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.566968    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0181d82d-82d2-4dd5-836c-6377670bd2c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0181d82d-82d2-4dd5-836c-6377670bd2c1" (UID: "0181d82d-82d2-4dd5-836c-6377670bd2c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:24.664473 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.664371    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:24.664473 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.664412    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:24.664473 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.664426    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8xml\" (UniqueName: \"kubernetes.io/projected/0181d82d-82d2-4dd5-836c-6377670bd2c1-kube-api-access-h8xml\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:24.664473 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.664439    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0181d82d-82d2-4dd5-836c-6377670bd2c1-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:24.664473 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.664450    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:24.664473 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.664461    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0181d82d-82d2-4dd5-836c-6377670bd2c1-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:24.899121 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.899085    2555 generic.go:356] "Generic (PLEG): container finished" podID="0181d82d-82d2-4dd5-836c-6377670bd2c1" containerID="a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5" exitCode=0
Sep 17 15:57:24.899121 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.899123    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2" event={"ID":"0181d82d-82d2-4dd5-836c-6377670bd2c1","Type":"ContainerDied","Data":"a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5"}
Sep 17 15:57:24.899311 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.899183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2" event={"ID":"0181d82d-82d2-4dd5-836c-6377670bd2c1","Type":"ContainerDied","Data":"e2ea9a5550c3a47c7e07117cb68484234367332f18e98e653bcc98158de8412e"}
Sep 17 15:57:24.899311 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.899205    2555 scope.go:117] "RemoveContainer" containerID="a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5"
Sep 17 15:57:24.899377 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.899326    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"
Sep 17 15:57:24.907946 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.907929    2555 scope.go:117] "RemoveContainer" containerID="a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5"
Sep 17 15:57:24.908222 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:57:24.908200    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5\": container with ID starting with a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5 not found: ID does not exist" containerID="a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5"
Sep 17 15:57:24.908294 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.908233    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5"} err="failed to get container status \"a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5\": rpc error: code = NotFound desc = could not find container \"a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5\": container with ID starting with a12b62a7702902e152958ecbd1078d6fe728d9db0bcaa9ebb60211ef23728ed5 not found: ID does not exist"
Sep 17 15:57:24.926755 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.926690    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"]
Sep 17 15:57:24.933289 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:24.933264    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-ltpb2"]
Sep 17 15:57:25.984584 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:25.984552    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"]
Sep 17 15:57:25.985156 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:25.984771    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll" podUID="99d31984-95cc-4dbb-9468-e87d684e19ca" containerName="istio-proxy" containerID="cri-o://fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a" gracePeriod=30
Sep 17 15:57:26.696315 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:26.696277    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0181d82d-82d2-4dd5-836c-6377670bd2c1" path="/var/lib/kubelet/pods/0181d82d-82d2-4dd5-836c-6377670bd2c1/volumes"
Sep 17 15:57:28.229904 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.229878    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:57:28.290274 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290209    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-podinfo\") pod \"99d31984-95cc-4dbb-9468-e87d684e19ca\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") "
Sep 17 15:57:28.290274 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290257    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-data\") pod \"99d31984-95cc-4dbb-9468-e87d684e19ca\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") "
Sep 17 15:57:28.290481 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290292    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-token\") pod \"99d31984-95cc-4dbb-9468-e87d684e19ca\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") "
Sep 17 15:57:28.290630 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290505    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82rdw\" (UniqueName: \"kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-kube-api-access-82rdw\") pod \"99d31984-95cc-4dbb-9468-e87d684e19ca\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") "
Sep 17 15:57:28.290630 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290567    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d31984-95cc-4dbb-9468-e87d684e19ca-istiod-ca-cert\") pod \"99d31984-95cc-4dbb-9468-e87d684e19ca\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") "
Sep 17 15:57:28.290630 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290591    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "99d31984-95cc-4dbb-9468-e87d684e19ca" (UID: "99d31984-95cc-4dbb-9468-e87d684e19ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:28.290630 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290601    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-certs\") pod \"99d31984-95cc-4dbb-9468-e87d684e19ca\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") "
Sep 17 15:57:28.290630 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290626    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-credential-socket\") pod \"99d31984-95cc-4dbb-9468-e87d684e19ca\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") "
Sep 17 15:57:28.290921 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290670    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-envoy\") pod \"99d31984-95cc-4dbb-9468-e87d684e19ca\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") "
Sep 17 15:57:28.290921 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290699    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-socket\") pod \"99d31984-95cc-4dbb-9468-e87d684e19ca\" (UID: \"99d31984-95cc-4dbb-9468-e87d684e19ca\") "
Sep 17 15:57:28.291032 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.290935    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d31984-95cc-4dbb-9468-e87d684e19ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99d31984-95cc-4dbb-9468-e87d684e19ca" (UID: "99d31984-95cc-4dbb-9468-e87d684e19ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:28.291134 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.291112    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d31984-95cc-4dbb-9468-e87d684e19ca-istiod-ca-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:28.291409 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.291388    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-data\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:28.291409 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.291193    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99d31984-95cc-4dbb-9468-e87d684e19ca" (UID: "99d31984-95cc-4dbb-9468-e87d684e19ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:28.291549 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.291214    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99d31984-95cc-4dbb-9468-e87d684e19ca" (UID: "99d31984-95cc-4dbb-9468-e87d684e19ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:28.291549 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.291271    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99d31984-95cc-4dbb-9468-e87d684e19ca" (UID: "99d31984-95cc-4dbb-9468-e87d684e19ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:28.292806 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.292786    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99d31984-95cc-4dbb-9468-e87d684e19ca" (UID: "99d31984-95cc-4dbb-9468-e87d684e19ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:28.293027 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.293004    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "99d31984-95cc-4dbb-9468-e87d684e19ca" (UID: "99d31984-95cc-4dbb-9468-e87d684e19ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:28.293291 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.293272    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-kube-api-access-82rdw" (OuterVolumeSpecName: "kube-api-access-82rdw") pod "99d31984-95cc-4dbb-9468-e87d684e19ca" (UID: "99d31984-95cc-4dbb-9468-e87d684e19ca"). InnerVolumeSpecName "kube-api-access-82rdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:28.294853 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.294815    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99d31984-95cc-4dbb-9468-e87d684e19ca" (UID: "99d31984-95cc-4dbb-9468-e87d684e19ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:57:28.392465 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.392443    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-token\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:28.392465 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.392465    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82rdw\" (UniqueName: \"kubernetes.io/projected/99d31984-95cc-4dbb-9468-e87d684e19ca-kube-api-access-82rdw\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:28.392598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.392475    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:28.392598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.392484    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-credential-socket\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:28.392598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.392492    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-envoy\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:28.392598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.392500    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d31984-95cc-4dbb-9468-e87d684e19ca-workload-socket\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:28.392598 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.392508    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d31984-95cc-4dbb-9468-e87d684e19ca-istio-podinfo\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:28.922013 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.921982    2555 generic.go:356] "Generic (PLEG): container finished" podID="99d31984-95cc-4dbb-9468-e87d684e19ca" containerID="fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a" exitCode=0
Sep 17 15:57:28.922182 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.922025    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"
Sep 17 15:57:28.922182 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.922030    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll" event={"ID":"99d31984-95cc-4dbb-9468-e87d684e19ca","Type":"ContainerDied","Data":"fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a"}
Sep 17 15:57:28.922182 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.922142    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll" event={"ID":"99d31984-95cc-4dbb-9468-e87d684e19ca","Type":"ContainerDied","Data":"2d47de5544b11048e61aa09c4b376bceb54a5dc3ba6ef5c4c12b11a9d4f2d8fa"}
Sep 17 15:57:28.922182 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.922163    2555 scope.go:117] "RemoveContainer" containerID="fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a"
Sep 17 15:57:28.930494 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.930477    2555 scope.go:117] "RemoveContainer" containerID="fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a"
Sep 17 15:57:28.930742 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:57:28.930722    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a\": container with ID starting with fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a not found: ID does not exist" containerID="fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a"
Sep 17 15:57:28.930789 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.930750    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a"} err="failed to get container status \"fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a\": rpc error: code = NotFound desc = could not find container \"fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a\": container with ID starting with fc4b82f9e1b88d683b1f5ab1fb3a201151da5a6b207a3e9dfc486d57b1f4459a not found: ID does not exist"
Sep 17 15:57:28.948468 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.948446    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"]
Sep 17 15:57:28.956041 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:28.955987    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-4dnll"]
Sep 17 15:57:30.694025 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:30.693993    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d31984-95cc-4dbb-9468-e87d684e19ca" path="/var/lib/kubelet/pods/99d31984-95cc-4dbb-9468-e87d684e19ca/volumes"
Sep 17 15:57:46.482563 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.482520    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"]
Sep 17 15:57:46.483082 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.483059    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d31984-95cc-4dbb-9468-e87d684e19ca" containerName="istio-proxy"
Sep 17 15:57:46.483129 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.483087    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d31984-95cc-4dbb-9468-e87d684e19ca" containerName="istio-proxy"
Sep 17 15:57:46.483129 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.483103    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0181d82d-82d2-4dd5-836c-6377670bd2c1" containerName="main"
Sep 17 15:57:46.483129 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.483112    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0181d82d-82d2-4dd5-836c-6377670bd2c1" containerName="main"
Sep 17 15:57:46.483224 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.483199    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d31984-95cc-4dbb-9468-e87d684e19ca" containerName="istio-proxy"
Sep 17 15:57:46.483261 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.483230    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0181d82d-82d2-4dd5-836c-6377670bd2c1" containerName="main"
Sep 17 15:57:46.486323 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.486297    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.490309 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.490280    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 15:57:46.490455 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.490314    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-l6dcf\""
Sep 17 15:57:46.490455 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.490329    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 15:57:46.490640 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.490626    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 15:57:46.495726 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.495702    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"]
Sep 17 15:57:46.524505 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.524477    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.524622 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.524507    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54g9q\" (UniqueName: \"kubernetes.io/projected/30711cc1-2a10-404e-9769-35845f736cb7-kube-api-access-54g9q\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.524622 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.524528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.524622 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.524591    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.524793 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.524638    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30711cc1-2a10-404e-9769-35845f736cb7-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.524793 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.524695    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.625975 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.625948    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.626090 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.625979    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54g9q\" (UniqueName: \"kubernetes.io/projected/30711cc1-2a10-404e-9769-35845f736cb7-kube-api-access-54g9q\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.626090 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.625998    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.626090 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.626028    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.626090 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.626055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30711cc1-2a10-404e-9769-35845f736cb7-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.626317 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.626098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.626404 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.626380    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.626494 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.626476    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.626554 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.626498    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.628236 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.628212    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.628399 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.628382    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30711cc1-2a10-404e-9769-35845f736cb7-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.634650 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.634613    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54g9q\" (UniqueName: \"kubernetes.io/projected/30711cc1-2a10-404e-9769-35845f736cb7-kube-api-access-54g9q\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:46.798918 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:46.798845    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:47.077496 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:47.077474    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"]
Sep 17 15:57:47.082690 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:57:47.082658    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30711cc1_2a10_404e_9769_35845f736cb7.slice/crio-9c0e918aef06a8457ae027d47c9a29de6b15433f48ea4900d945e8e529a26e67 WatchSource:0}: Error finding container 9c0e918aef06a8457ae027d47c9a29de6b15433f48ea4900d945e8e529a26e67: Status 404 returned error can't find the container with id 9c0e918aef06a8457ae027d47c9a29de6b15433f48ea4900d945e8e529a26e67
Sep 17 15:57:48.016932 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:48.016888    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc" event={"ID":"30711cc1-2a10-404e-9769-35845f736cb7","Type":"ContainerStarted","Data":"d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184"}
Sep 17 15:57:48.017394 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:48.016972    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc" event={"ID":"30711cc1-2a10-404e-9769-35845f736cb7","Type":"ContainerStarted","Data":"9c0e918aef06a8457ae027d47c9a29de6b15433f48ea4900d945e8e529a26e67"}
Sep 17 15:57:48.044843 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:48.044776    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc" podStartSLOduration=2.04476053 podStartE2EDuration="2.04476053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:48.04398728 +0000 UTC m=+1149.934847825" watchObservedRunningTime="2026-09-17 15:57:48.04476053 +0000 UTC m=+1149.935621074"
Sep 17 15:57:48.799365 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:48.799338    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:50.851391 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:50.851350    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"]
Sep 17 15:57:51.040272 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:57:51.040235    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 15:57:51.040272 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:57:51.040272 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:57:51.040272 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184" cmd=["/bin/sleep","15"]
Sep 17 15:57:51.040517 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.040281    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc" message="" err=<
Sep 17 15:57:51.040517 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:57:51.040517 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:57:51.040517 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 15:57:51.040517 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:57:51.040292    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 15:57:51.040517 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:57:51.040517 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:57:51.040517 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc" podUID="30711cc1-2a10-404e-9769-35845f736cb7" containerName="main" containerID="cri-o://d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184"
Sep 17 15:57:51.040517 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.040327    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc" podUID="30711cc1-2a10-404e-9769-35845f736cb7" containerName="main" containerID="cri-o://d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184" gracePeriod=60
Sep 17 15:57:51.282294 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.282271    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:51.363951 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.363925    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-home\") pod \"30711cc1-2a10-404e-9769-35845f736cb7\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") "
Sep 17 15:57:51.364092 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.363968    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-tmp-dir\") pod \"30711cc1-2a10-404e-9769-35845f736cb7\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") "
Sep 17 15:57:51.364092 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.363989    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-dshm\") pod \"30711cc1-2a10-404e-9769-35845f736cb7\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") "
Sep 17 15:57:51.364092 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.364040    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30711cc1-2a10-404e-9769-35845f736cb7-tls-certs\") pod \"30711cc1-2a10-404e-9769-35845f736cb7\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") "
Sep 17 15:57:51.364092 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.364076    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54g9q\" (UniqueName: \"kubernetes.io/projected/30711cc1-2a10-404e-9769-35845f736cb7-kube-api-access-54g9q\") pod \"30711cc1-2a10-404e-9769-35845f736cb7\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") "
Sep 17 15:57:51.364256 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.364122    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-model-cache\") pod \"30711cc1-2a10-404e-9769-35845f736cb7\" (UID: \"30711cc1-2a10-404e-9769-35845f736cb7\") "
Sep 17 15:57:51.364256 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.364217    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30711cc1-2a10-404e-9769-35845f736cb7" (UID: "30711cc1-2a10-404e-9769-35845f736cb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:51.364349 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.364230    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-home" (OuterVolumeSpecName: "home") pod "30711cc1-2a10-404e-9769-35845f736cb7" (UID: "30711cc1-2a10-404e-9769-35845f736cb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:51.364389 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.364376    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:51.364435 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.364397    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:51.364435 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.364404    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-model-cache" (OuterVolumeSpecName: "model-cache") pod "30711cc1-2a10-404e-9769-35845f736cb7" (UID: "30711cc1-2a10-404e-9769-35845f736cb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:51.366189 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.366159    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-dshm" (OuterVolumeSpecName: "dshm") pod "30711cc1-2a10-404e-9769-35845f736cb7" (UID: "30711cc1-2a10-404e-9769-35845f736cb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:51.366302 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.366224    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30711cc1-2a10-404e-9769-35845f736cb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30711cc1-2a10-404e-9769-35845f736cb7" (UID: "30711cc1-2a10-404e-9769-35845f736cb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:51.366343 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.366306    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30711cc1-2a10-404e-9769-35845f736cb7-kube-api-access-54g9q" (OuterVolumeSpecName: "kube-api-access-54g9q") pod "30711cc1-2a10-404e-9769-35845f736cb7" (UID: "30711cc1-2a10-404e-9769-35845f736cb7"). InnerVolumeSpecName "kube-api-access-54g9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:51.465913 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.465815    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:51.465913 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.465870    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30711cc1-2a10-404e-9769-35845f736cb7-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:51.465913 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.465886    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54g9q\" (UniqueName: \"kubernetes.io/projected/30711cc1-2a10-404e-9769-35845f736cb7-kube-api-access-54g9q\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:51.465913 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:51.465901    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30711cc1-2a10-404e-9769-35845f736cb7-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:57:52.038266 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:52.038237    2555 generic.go:356] "Generic (PLEG): container finished" podID="30711cc1-2a10-404e-9769-35845f736cb7" containerID="d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184" exitCode=0
Sep 17 15:57:52.038682 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:52.038281    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"
Sep 17 15:57:52.038682 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:52.038323    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc" event={"ID":"30711cc1-2a10-404e-9769-35845f736cb7","Type":"ContainerDied","Data":"d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184"}
Sep 17 15:57:52.038682 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:52.038375    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc" event={"ID":"30711cc1-2a10-404e-9769-35845f736cb7","Type":"ContainerDied","Data":"9c0e918aef06a8457ae027d47c9a29de6b15433f48ea4900d945e8e529a26e67"}
Sep 17 15:57:52.038682 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:52.038399    2555 scope.go:117] "RemoveContainer" containerID="d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184"
Sep 17 15:57:52.047668 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:52.047524    2555 scope.go:117] "RemoveContainer" containerID="d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184"
Sep 17 15:57:52.047850 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:57:52.047808    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184\": container with ID starting with d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184 not found: ID does not exist" containerID="d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184"
Sep 17 15:57:52.047902 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:52.047860    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184"} err="failed to get container status \"d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184\": rpc error: code = NotFound desc = could not find container \"d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184\": container with ID starting with d3e5589176c0293eb4de1afc5411b75c4447226725097264f6677ee3afa61184 not found: ID does not exist"
Sep 17 15:57:52.061184 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:52.061163    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"]
Sep 17 15:57:52.063519 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:52.063498    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdt2zc"]
Sep 17 15:57:52.699993 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:57:52.695634    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30711cc1-2a10-404e-9769-35845f736cb7" path="/var/lib/kubelet/pods/30711cc1-2a10-404e-9769-35845f736cb7/volumes"
Sep 17 15:58:07.912098 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.912064    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"]
Sep 17 15:58:07.912493 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.912365    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30711cc1-2a10-404e-9769-35845f736cb7" containerName="main"
Sep 17 15:58:07.912493 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.912376    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30711cc1-2a10-404e-9769-35845f736cb7" containerName="main"
Sep 17 15:58:07.912493 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.912426    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30711cc1-2a10-404e-9769-35845f736cb7" containerName="main"
Sep 17 15:58:07.914636 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.914619    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 15:58:07.917414 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.917392    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 15:58:07.917544 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.917467    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 15:58:07.918669 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.918652    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-2xcpn\""
Sep 17 15:58:07.918775 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.918676    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-2z546\""
Sep 17 15:58:07.918775 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.918758    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 15:58:07.926833 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.926803    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"]
Sep 17 15:58:07.978050 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.978026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b597\" (UniqueName: \"kubernetes.io/projected/3f805792-21d2-48c6-a534-5829abfb55fa-kube-api-access-2b597\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7\" (UID: \"3f805792-21d2-48c6-a534-5829abfb55fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 15:58:07.978134 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:07.978105    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f805792-21d2-48c6-a534-5829abfb55fa-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7\" (UID: \"3f805792-21d2-48c6-a534-5829abfb55fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 15:58:08.078971 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:08.078938    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b597\" (UniqueName: \"kubernetes.io/projected/3f805792-21d2-48c6-a534-5829abfb55fa-kube-api-access-2b597\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7\" (UID: \"3f805792-21d2-48c6-a534-5829abfb55fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 15:58:08.079081 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:08.079002    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f805792-21d2-48c6-a534-5829abfb55fa-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7\" (UID: \"3f805792-21d2-48c6-a534-5829abfb55fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 15:58:08.081247 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:08.081228    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f805792-21d2-48c6-a534-5829abfb55fa-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7\" (UID: \"3f805792-21d2-48c6-a534-5829abfb55fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 15:58:08.087418 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:08.087393    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b597\" (UniqueName: \"kubernetes.io/projected/3f805792-21d2-48c6-a534-5829abfb55fa-kube-api-access-2b597\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7\" (UID: \"3f805792-21d2-48c6-a534-5829abfb55fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 15:58:08.225048 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:08.224987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 15:58:08.494452 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:08.494424    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"]
Sep 17 15:58:08.496277 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:58:08.496249    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f805792_21d2_48c6_a534_5829abfb55fa.slice/crio-9b35db19cc4533326fed32859640d9648e7cf88ee43dd8fe351a19381f5f07b3 WatchSource:0}: Error finding container 9b35db19cc4533326fed32859640d9648e7cf88ee43dd8fe351a19381f5f07b3: Status 404 returned error can't find the container with id 9b35db19cc4533326fed32859640d9648e7cf88ee43dd8fe351a19381f5f07b3
Sep 17 15:58:09.120162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:09.120126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7" event={"ID":"3f805792-21d2-48c6-a534-5829abfb55fa","Type":"ContainerStarted","Data":"4dcf87467aa07e4c8a846cf5c6a0816a6359371c3fe5f5e94b6b219f231fab07"}
Sep 17 15:58:09.120642 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:09.120185    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7" event={"ID":"3f805792-21d2-48c6-a534-5829abfb55fa","Type":"ContainerStarted","Data":"9b35db19cc4533326fed32859640d9648e7cf88ee43dd8fe351a19381f5f07b3"}
Sep 17 15:58:09.120642 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:09.120276    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 15:58:09.139746 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:09.139695    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7" podStartSLOduration=2.139680002 podStartE2EDuration="2.139680002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:58:09.137566739 +0000 UTC m=+1171.028427298" watchObservedRunningTime="2026-09-17 15:58:09.139680002 +0000 UTC m=+1171.030540545"
Sep 17 15:58:38.724673 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:38.724644    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:58:38.725179 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:38.725126    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 15:58:40.126848 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:40.126796    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 15:58:57.357460 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.357427    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"]
Sep 17 15:58:57.366156 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:58:57.366118    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 15:58:57.366156 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:58:57.366156 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:58:57.366156 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec" cmd=["/bin/sleep","15"]
Sep 17 15:58:57.366400 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.366168    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp" message="" err=<
Sep 17 15:58:57.366400 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:58:57.366400 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:58:57.366400 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 15:58:57.366400 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:58:57.366179    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 15:58:57.366400 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:58:57.366400 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 15:58:57.366400 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp" podUID="f744659a-f38b-4316-8de3-7d717891a34c" containerName="main" containerID="cri-o://71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec"
Sep 17 15:58:57.366400 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.366226    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp" podUID="f744659a-f38b-4316-8de3-7d717891a34c" containerName="main" containerID="cri-o://71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec" gracePeriod=60
Sep 17 15:58:57.699914 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.699888    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:58:57.751680 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.751655    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-tmp-dir\") pod \"f744659a-f38b-4316-8de3-7d717891a34c\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") "
Sep 17 15:58:57.751809 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.751715    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl888\" (UniqueName: \"kubernetes.io/projected/f744659a-f38b-4316-8de3-7d717891a34c-kube-api-access-tl888\") pod \"f744659a-f38b-4316-8de3-7d717891a34c\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") "
Sep 17 15:58:57.751809 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.751741    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-home\") pod \"f744659a-f38b-4316-8de3-7d717891a34c\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") "
Sep 17 15:58:57.751809 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.751788    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-dshm\") pod \"f744659a-f38b-4316-8de3-7d717891a34c\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") "
Sep 17 15:58:57.752004 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.751885    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-model-cache\") pod \"f744659a-f38b-4316-8de3-7d717891a34c\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") "
Sep 17 15:58:57.752004 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.751909    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f744659a-f38b-4316-8de3-7d717891a34c" (UID: "f744659a-f38b-4316-8de3-7d717891a34c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:57.752004 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.751928    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f744659a-f38b-4316-8de3-7d717891a34c-tls-certs\") pod \"f744659a-f38b-4316-8de3-7d717891a34c\" (UID: \"f744659a-f38b-4316-8de3-7d717891a34c\") "
Sep 17 15:58:57.752162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.752022    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-home" (OuterVolumeSpecName: "home") pod "f744659a-f38b-4316-8de3-7d717891a34c" (UID: "f744659a-f38b-4316-8de3-7d717891a34c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:57.752162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.752097    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f744659a-f38b-4316-8de3-7d717891a34c" (UID: "f744659a-f38b-4316-8de3-7d717891a34c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:57.752162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.752106    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:58:57.752162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.752123    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:58:57.754028 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.754004    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-dshm" (OuterVolumeSpecName: "dshm") pod "f744659a-f38b-4316-8de3-7d717891a34c" (UID: "f744659a-f38b-4316-8de3-7d717891a34c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:57.754126 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.754028    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f744659a-f38b-4316-8de3-7d717891a34c-kube-api-access-tl888" (OuterVolumeSpecName: "kube-api-access-tl888") pod "f744659a-f38b-4316-8de3-7d717891a34c" (UID: "f744659a-f38b-4316-8de3-7d717891a34c"). InnerVolumeSpecName "kube-api-access-tl888". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:58:57.754126 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.754100    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f744659a-f38b-4316-8de3-7d717891a34c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f744659a-f38b-4316-8de3-7d717891a34c" (UID: "f744659a-f38b-4316-8de3-7d717891a34c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:57.852451 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.852421    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl888\" (UniqueName: \"kubernetes.io/projected/f744659a-f38b-4316-8de3-7d717891a34c-kube-api-access-tl888\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:58:57.852451 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.852445    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:58:57.852451 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.852455    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f744659a-f38b-4316-8de3-7d717891a34c-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:58:57.852646 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:57.852464    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f744659a-f38b-4316-8de3-7d717891a34c-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 15:58:58.362462 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:58.362426    2555 generic.go:356] "Generic (PLEG): container finished" podID="f744659a-f38b-4316-8de3-7d717891a34c" containerID="71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec" exitCode=0
Sep 17 15:58:58.362884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:58.362470    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp" event={"ID":"f744659a-f38b-4316-8de3-7d717891a34c","Type":"ContainerDied","Data":"71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec"}
Sep 17 15:58:58.362884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:58.362488    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"
Sep 17 15:58:58.362884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:58.362553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp" event={"ID":"f744659a-f38b-4316-8de3-7d717891a34c","Type":"ContainerDied","Data":"ffae8a1a44fe33140c164dbbc1d1fe4789e08ac3ddb239822527d9361d5ad1ce"}
Sep 17 15:58:58.362884 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:58.362586    2555 scope.go:117] "RemoveContainer" containerID="71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec"
Sep 17 15:58:58.371925 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:58.371713    2555 scope.go:117] "RemoveContainer" containerID="71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec"
Sep 17 15:58:58.372024 ip-10-0-135-243 kubenswrapper[2555]: E0917 15:58:58.372002    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec\": container with ID starting with 71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec not found: ID does not exist" containerID="71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec"
Sep 17 15:58:58.372066 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:58.372032    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec"} err="failed to get container status \"71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec\": rpc error: code = NotFound desc = could not find container \"71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec\": container with ID starting with 71ee411c1ec1fdaa69ad3f3e85ca204b870ee4b0dfa9c6b780f8a81072626bec not found: ID does not exist"
Sep 17 15:58:58.385162 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:58.385135    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"]
Sep 17 15:58:58.390129 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:58.390102    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-2j4kp"]
Sep 17 15:58:58.695383 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:58:58.695289    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f744659a-f38b-4316-8de3-7d717891a34c" path="/var/lib/kubelet/pods/f744659a-f38b-4316-8de3-7d717891a34c/volumes"
Sep 17 15:59:54.334561 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.334528    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"]
Sep 17 15:59:54.335025 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.334833    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f744659a-f38b-4316-8de3-7d717891a34c" containerName="main"
Sep 17 15:59:54.335025 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.334849    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f744659a-f38b-4316-8de3-7d717891a34c" containerName="main"
Sep 17 15:59:54.335025 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.334913    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f744659a-f38b-4316-8de3-7d717891a34c" containerName="main"
Sep 17 15:59:54.338990 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.338968    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.341930 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.341906    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 15:59:54.343197 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.343175    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-7pqfk\""
Sep 17 15:59:54.343289 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.343202    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 15:59:54.343289 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.343274    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 15:59:54.350163 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.350144    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"]
Sep 17 15:59:54.441336 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.441314    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.441456 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.441358    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.441456 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.441376    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.441456 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.441399    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djs9d\" (UniqueName: \"kubernetes.io/projected/b8754ba9-3cf6-49a7-8983-3b1107bcc277-kube-api-access-djs9d\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.441456 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.441454    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.441628 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.441481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.542112 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.542092    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.542239 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.542126    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.542239 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.542143    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.542239 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.542162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djs9d\" (UniqueName: \"kubernetes.io/projected/b8754ba9-3cf6-49a7-8983-3b1107bcc277-kube-api-access-djs9d\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.542239 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.542196    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.542239 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.542235    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.542556 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.542542    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.542617 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.542576    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.542670 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.542619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.544473 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.544452    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.544761 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.544744    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.550942 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.550919    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djs9d\" (UniqueName: \"kubernetes.io/projected/b8754ba9-3cf6-49a7-8983-3b1107bcc277-kube-api-access-djs9d\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.650295 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.650243    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:54.922278 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:54.922251    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"]
Sep 17 15:59:54.922925 ip-10-0-135-243 kubenswrapper[2555]: W0917 15:59:54.922899    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8754ba9_3cf6_49a7_8983_3b1107bcc277.slice/crio-68c40efbcc6bb594e0f142d6cf6153cde35006c757a962a170bcbae7854595ce WatchSource:0}: Error finding container 68c40efbcc6bb594e0f142d6cf6153cde35006c757a962a170bcbae7854595ce: Status 404 returned error can't find the container with id 68c40efbcc6bb594e0f142d6cf6153cde35006c757a962a170bcbae7854595ce
Sep 17 15:59:55.643058 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:55.643025    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb" event={"ID":"b8754ba9-3cf6-49a7-8983-3b1107bcc277","Type":"ContainerStarted","Data":"847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b"}
Sep 17 15:59:55.643442 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:55.643076    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb" event={"ID":"b8754ba9-3cf6-49a7-8983-3b1107bcc277","Type":"ContainerStarted","Data":"68c40efbcc6bb594e0f142d6cf6153cde35006c757a962a170bcbae7854595ce"}
Sep 17 15:59:55.650408 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:55.650389    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 15:59:55.668647 ip-10-0-135-243 kubenswrapper[2555]: I0917 15:59:55.668609    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb" podStartSLOduration=1.668595733 podStartE2EDuration="1.668595733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:55.667038658 +0000 UTC m=+1277.557899201" watchObservedRunningTime="2026-09-17 15:59:55.668595733 +0000 UTC m=+1277.559456278"
Sep 17 16:00:04.651002 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:04.650960    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 16:00:04.663367 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:04.663336    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 16:00:04.705074 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:04.705049    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 16:00:29.243655 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.243621    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"]
Sep 17 16:00:29.252447 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:00:29.252408    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:00:29.252447 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:00:29.252447 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:00:29.252447 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b" cmd=["/bin/sleep","15"]
Sep 17 16:00:29.252707 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.252469    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb" message="" err=<
Sep 17 16:00:29.252707 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:00:29.252707 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:00:29.252707 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 16:00:29.252707 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:00:29.252488    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:00:29.252707 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:00:29.252707 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:00:29.252707 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb" podUID="b8754ba9-3cf6-49a7-8983-3b1107bcc277" containerName="main" containerID="cri-o://847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b"
Sep 17 16:00:29.252707 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.252531    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb" podUID="b8754ba9-3cf6-49a7-8983-3b1107bcc277" containerName="main" containerID="cri-o://847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b" gracePeriod=60
Sep 17 16:00:29.596885 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.596863    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 16:00:29.693779 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.693754    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-home\") pod \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") "
Sep 17 16:00:29.693902 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.693804    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-model-cache\") pod \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") "
Sep 17 16:00:29.693902 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.693865    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-dshm\") pod \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") "
Sep 17 16:00:29.693902 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.693894    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tls-certs\") pod \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") "
Sep 17 16:00:29.694068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.693928    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tmp-dir\") pod \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") "
Sep 17 16:00:29.694068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.693965    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djs9d\" (UniqueName: \"kubernetes.io/projected/b8754ba9-3cf6-49a7-8983-3b1107bcc277-kube-api-access-djs9d\") pod \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\" (UID: \"b8754ba9-3cf6-49a7-8983-3b1107bcc277\") "
Sep 17 16:00:29.694175 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.694047    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-home" (OuterVolumeSpecName: "home") pod "b8754ba9-3cf6-49a7-8983-3b1107bcc277" (UID: "b8754ba9-3cf6-49a7-8983-3b1107bcc277"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:29.694175 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.694092    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8754ba9-3cf6-49a7-8983-3b1107bcc277" (UID: "b8754ba9-3cf6-49a7-8983-3b1107bcc277"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:29.694284 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.694181    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.694284 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.694199    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.694365 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.694287    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8754ba9-3cf6-49a7-8983-3b1107bcc277" (UID: "b8754ba9-3cf6-49a7-8983-3b1107bcc277"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:29.696121 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.696095    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8754ba9-3cf6-49a7-8983-3b1107bcc277" (UID: "b8754ba9-3cf6-49a7-8983-3b1107bcc277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:00:29.696213 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.696159    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8754ba9-3cf6-49a7-8983-3b1107bcc277-kube-api-access-djs9d" (OuterVolumeSpecName: "kube-api-access-djs9d") pod "b8754ba9-3cf6-49a7-8983-3b1107bcc277" (UID: "b8754ba9-3cf6-49a7-8983-3b1107bcc277"). InnerVolumeSpecName "kube-api-access-djs9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:00:29.696213 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.696189    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-dshm" (OuterVolumeSpecName: "dshm") pod "b8754ba9-3cf6-49a7-8983-3b1107bcc277" (UID: "b8754ba9-3cf6-49a7-8983-3b1107bcc277"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:29.795304 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.795250    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.795304 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.795270    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djs9d\" (UniqueName: \"kubernetes.io/projected/b8754ba9-3cf6-49a7-8983-3b1107bcc277-kube-api-access-djs9d\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.795304 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.795283    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8754ba9-3cf6-49a7-8983-3b1107bcc277-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.795304 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.795295    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8754ba9-3cf6-49a7-8983-3b1107bcc277-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.812016 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.811989    2555 generic.go:356] "Generic (PLEG): container finished" podID="b8754ba9-3cf6-49a7-8983-3b1107bcc277" containerID="847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b" exitCode=0
Sep 17 16:00:29.812106 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.812037    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"
Sep 17 16:00:29.812151 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.812036    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb" event={"ID":"b8754ba9-3cf6-49a7-8983-3b1107bcc277","Type":"ContainerDied","Data":"847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b"}
Sep 17 16:00:29.812187 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.812153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb" event={"ID":"b8754ba9-3cf6-49a7-8983-3b1107bcc277","Type":"ContainerDied","Data":"68c40efbcc6bb594e0f142d6cf6153cde35006c757a962a170bcbae7854595ce"}
Sep 17 16:00:29.812187 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.812183    2555 scope.go:117] "RemoveContainer" containerID="847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b"
Sep 17 16:00:29.821015 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.820999    2555 scope.go:117] "RemoveContainer" containerID="847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b"
Sep 17 16:00:29.821272 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:00:29.821255    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b\": container with ID starting with 847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b not found: ID does not exist" containerID="847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b"
Sep 17 16:00:29.821321 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.821283    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b"} err="failed to get container status \"847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b\": rpc error: code = NotFound desc = could not find container \"847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b\": container with ID starting with 847d8e6b06a31f55172cd39823b454bbc54f084a787af8696758e197a693342b not found: ID does not exist"
Sep 17 16:00:29.834318 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.834295    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"]
Sep 17 16:00:29.837628 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:29.837603    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mb7gb"]
Sep 17 16:00:30.694290 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:30.694253    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8754ba9-3cf6-49a7-8983-3b1107bcc277" path="/var/lib/kubelet/pods/b8754ba9-3cf6-49a7-8983-3b1107bcc277/volumes"
Sep 17 16:00:42.891988 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:42.891953    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"]
Sep 17 16:00:50.126323 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:50.126281    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7" podUID="3f805792-21d2-48c6-a534-5829abfb55fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:00:57.911002 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:00:57.910943    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7" podUID="3f805792-21d2-48c6-a534-5829abfb55fa" containerName="main" containerID="cri-o://4dcf87467aa07e4c8a846cf5c6a0816a6359371c3fe5f5e94b6b219f231fab07" gracePeriod=45
Sep 17 16:01:00.126604 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:00.126569    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7" podUID="3f805792-21d2-48c6-a534-5829abfb55fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:01:10.126673 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:10.126633    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7" podUID="3f805792-21d2-48c6-a534-5829abfb55fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:01:10.127236 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:10.126735    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 16:01:20.125723 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:20.125679    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7" podUID="3f805792-21d2-48c6-a534-5829abfb55fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:01:23.073079 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.073048    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"]
Sep 17 16:01:23.073574 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.073356    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8754ba9-3cf6-49a7-8983-3b1107bcc277" containerName="main"
Sep 17 16:01:23.073574 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.073370    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8754ba9-3cf6-49a7-8983-3b1107bcc277" containerName="main"
Sep 17 16:01:23.073574 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.073425    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8754ba9-3cf6-49a7-8983-3b1107bcc277" containerName="main"
Sep 17 16:01:23.076189 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.076173    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:01:23.080357 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.080330    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-jrgx7\""
Sep 17 16:01:23.080498 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.080360    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 16:01:23.080498 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.080376    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-gplt5\""
Sep 17 16:01:23.080498 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.080396    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 16:01:23.080498 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.080370    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 16:01:23.087459 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.087438    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"]
Sep 17 16:01:23.157168 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.157147    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9z99\" (UniqueName: \"kubernetes.io/projected/a52de307-2a13-4669-8c52-5d53a8d00e93-kube-api-access-j9z99\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj\" (UID: \"a52de307-2a13-4669-8c52-5d53a8d00e93\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:01:23.157276 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.157205    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52de307-2a13-4669-8c52-5d53a8d00e93-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj\" (UID: \"a52de307-2a13-4669-8c52-5d53a8d00e93\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:01:23.258068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.258047    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9z99\" (UniqueName: \"kubernetes.io/projected/a52de307-2a13-4669-8c52-5d53a8d00e93-kube-api-access-j9z99\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj\" (UID: \"a52de307-2a13-4669-8c52-5d53a8d00e93\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:01:23.258168 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.258132    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52de307-2a13-4669-8c52-5d53a8d00e93-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj\" (UID: \"a52de307-2a13-4669-8c52-5d53a8d00e93\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:01:23.260628 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.260601    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52de307-2a13-4669-8c52-5d53a8d00e93-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj\" (UID: \"a52de307-2a13-4669-8c52-5d53a8d00e93\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:01:23.266734 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.266715    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9z99\" (UniqueName: \"kubernetes.io/projected/a52de307-2a13-4669-8c52-5d53a8d00e93-kube-api-access-j9z99\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj\" (UID: \"a52de307-2a13-4669-8c52-5d53a8d00e93\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:01:23.386719 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.386697    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:01:23.664296 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:23.663979    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"]
Sep 17 16:01:23.667539 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:01:23.667283    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52de307_2a13_4669_8c52_5d53a8d00e93.slice/crio-42c4e5aa52c35abf14e9d24460e4240d473d8aa70049be3790f65c79d9221945 WatchSource:0}: Error finding container 42c4e5aa52c35abf14e9d24460e4240d473d8aa70049be3790f65c79d9221945: Status 404 returned error can't find the container with id 42c4e5aa52c35abf14e9d24460e4240d473d8aa70049be3790f65c79d9221945
Sep 17 16:01:24.069161 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:24.069062    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj" event={"ID":"a52de307-2a13-4669-8c52-5d53a8d00e93","Type":"ContainerStarted","Data":"b66bf921ecc581d946d6f11fc8cdde14dfd347923811822db65a43015d15e04b"}
Sep 17 16:01:24.069161 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:24.069125    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj" event={"ID":"a52de307-2a13-4669-8c52-5d53a8d00e93","Type":"ContainerStarted","Data":"42c4e5aa52c35abf14e9d24460e4240d473d8aa70049be3790f65c79d9221945"}
Sep 17 16:01:24.069366 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:24.069243    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:01:24.094931 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:24.094875    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj" podStartSLOduration=1.09485649 podStartE2EDuration="1.09485649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:24.093393227 +0000 UTC m=+1365.984253784" watchObservedRunningTime="2026-09-17 16:01:24.09485649 +0000 UTC m=+1365.985717035"
Sep 17 16:01:28.092260 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:28.092233    2555 generic.go:356] "Generic (PLEG): container finished" podID="3f805792-21d2-48c6-a534-5829abfb55fa" containerID="4dcf87467aa07e4c8a846cf5c6a0816a6359371c3fe5f5e94b6b219f231fab07" exitCode=0
Sep 17 16:01:28.092576 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:28.092286    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7" event={"ID":"3f805792-21d2-48c6-a534-5829abfb55fa","Type":"ContainerDied","Data":"4dcf87467aa07e4c8a846cf5c6a0816a6359371c3fe5f5e94b6b219f231fab07"}
Sep 17 16:01:28.150682 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:28.150662    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 16:01:28.196757 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:28.196696    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f805792-21d2-48c6-a534-5829abfb55fa-tls-certs\") pod \"3f805792-21d2-48c6-a534-5829abfb55fa\" (UID: \"3f805792-21d2-48c6-a534-5829abfb55fa\") "
Sep 17 16:01:28.196871 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:28.196757    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b597\" (UniqueName: \"kubernetes.io/projected/3f805792-21d2-48c6-a534-5829abfb55fa-kube-api-access-2b597\") pod \"3f805792-21d2-48c6-a534-5829abfb55fa\" (UID: \"3f805792-21d2-48c6-a534-5829abfb55fa\") "
Sep 17 16:01:28.198873 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:28.198847    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f805792-21d2-48c6-a534-5829abfb55fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f805792-21d2-48c6-a534-5829abfb55fa" (UID: "3f805792-21d2-48c6-a534-5829abfb55fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:01:28.198977 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:28.198900    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f805792-21d2-48c6-a534-5829abfb55fa-kube-api-access-2b597" (OuterVolumeSpecName: "kube-api-access-2b597") pod "3f805792-21d2-48c6-a534-5829abfb55fa" (UID: "3f805792-21d2-48c6-a534-5829abfb55fa"). InnerVolumeSpecName "kube-api-access-2b597". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:28.297317 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:28.297297    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b597\" (UniqueName: \"kubernetes.io/projected/3f805792-21d2-48c6-a534-5829abfb55fa-kube-api-access-2b597\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:01:28.297317 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:28.297317    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f805792-21d2-48c6-a534-5829abfb55fa-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:01:29.097959 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:29.097930    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"
Sep 17 16:01:29.097959 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:29.097945    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7" event={"ID":"3f805792-21d2-48c6-a534-5829abfb55fa","Type":"ContainerDied","Data":"9b35db19cc4533326fed32859640d9648e7cf88ee43dd8fe351a19381f5f07b3"}
Sep 17 16:01:29.098446 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:29.098007    2555 scope.go:117] "RemoveContainer" containerID="4dcf87467aa07e4c8a846cf5c6a0816a6359371c3fe5f5e94b6b219f231fab07"
Sep 17 16:01:29.121862 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:29.117972    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"]
Sep 17 16:01:29.124311 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:29.124290    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wxlq7"]
Sep 17 16:01:30.693964 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:30.693932    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f805792-21d2-48c6-a534-5829abfb55fa" path="/var/lib/kubelet/pods/3f805792-21d2-48c6-a534-5829abfb55fa/volumes"
Sep 17 16:01:55.076043 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:55.076011    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:01:58.869502 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.869467    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"]
Sep 17 16:01:58.870011 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.869809    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f805792-21d2-48c6-a534-5829abfb55fa" containerName="main"
Sep 17 16:01:58.870011 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.869836    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f805792-21d2-48c6-a534-5829abfb55fa" containerName="main"
Sep 17 16:01:58.870011 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.869905    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f805792-21d2-48c6-a534-5829abfb55fa" containerName="main"
Sep 17 16:01:58.874178 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.874153    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:01:58.878471 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.878445    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-fbmzl\""
Sep 17 16:01:58.878595 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.878547    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 16:01:58.878668 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.878605    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 16:01:58.878731 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.878695    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-4cktr\""
Sep 17 16:01:58.878901 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.878880    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 16:01:58.885785 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:58.885760    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"]
Sep 17 16:01:59.012980 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:59.012947    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0849f0db-fd42-446e-aa02-31f1521657b3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd\" (UID: \"0849f0db-fd42-446e-aa02-31f1521657b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:01:59.012980 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:59.012984    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bstf\" (UniqueName: \"kubernetes.io/projected/0849f0db-fd42-446e-aa02-31f1521657b3-kube-api-access-2bstf\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd\" (UID: \"0849f0db-fd42-446e-aa02-31f1521657b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:01:59.113992 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:59.113962    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0849f0db-fd42-446e-aa02-31f1521657b3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd\" (UID: \"0849f0db-fd42-446e-aa02-31f1521657b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:01:59.114190 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:59.114008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bstf\" (UniqueName: \"kubernetes.io/projected/0849f0db-fd42-446e-aa02-31f1521657b3-kube-api-access-2bstf\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd\" (UID: \"0849f0db-fd42-446e-aa02-31f1521657b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:01:59.116388 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:59.116364    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0849f0db-fd42-446e-aa02-31f1521657b3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd\" (UID: \"0849f0db-fd42-446e-aa02-31f1521657b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:01:59.123863 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:59.123796    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bstf\" (UniqueName: \"kubernetes.io/projected/0849f0db-fd42-446e-aa02-31f1521657b3-kube-api-access-2bstf\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd\" (UID: \"0849f0db-fd42-446e-aa02-31f1521657b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:01:59.184747 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:59.184727    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:01:59.457986 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:01:59.457960    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"]
Sep 17 16:01:59.459815 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:01:59.459776    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0849f0db_fd42_446e_aa02_31f1521657b3.slice/crio-e5a0cfc57ef63d395a3a771781c8f395f5563272d3ec003f848adf6b7df9b257 WatchSource:0}: Error finding container e5a0cfc57ef63d395a3a771781c8f395f5563272d3ec003f848adf6b7df9b257: Status 404 returned error can't find the container with id e5a0cfc57ef63d395a3a771781c8f395f5563272d3ec003f848adf6b7df9b257
Sep 17 16:02:00.250463 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:02:00.250426    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd" event={"ID":"0849f0db-fd42-446e-aa02-31f1521657b3","Type":"ContainerStarted","Data":"8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836"}
Sep 17 16:02:00.250970 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:02:00.250478    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd" event={"ID":"0849f0db-fd42-446e-aa02-31f1521657b3","Type":"ContainerStarted","Data":"e5a0cfc57ef63d395a3a771781c8f395f5563272d3ec003f848adf6b7df9b257"}
Sep 17 16:02:00.250970 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:02:00.250500    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:02:00.270481 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:02:00.270435    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd" podStartSLOduration=2.2704175810000002 podStartE2EDuration="2.270417581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:02:00.269295982 +0000 UTC m=+1402.160156529" watchObservedRunningTime="2026-09-17 16:02:00.270417581 +0000 UTC m=+1402.161278125"
Sep 17 16:02:31.257212 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:02:31.257137    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:02:47.570537 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:02:47.570502    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"]
Sep 17 16:02:55.075842 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:02:55.075782    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj" podUID="a52de307-2a13-4669-8c52-5d53a8d00e93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:03:02.589190 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:02.589143    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj" podUID="a52de307-2a13-4669-8c52-5d53a8d00e93" containerName="main" containerID="cri-o://b66bf921ecc581d946d6f11fc8cdde14dfd347923811822db65a43015d15e04b" gracePeriod=45
Sep 17 16:03:05.076061 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:05.076026    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj" podUID="a52de307-2a13-4669-8c52-5d53a8d00e93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:03:15.076122 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:15.076076    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj" podUID="a52de307-2a13-4669-8c52-5d53a8d00e93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:03:15.076699 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:15.076188    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:03:25.075840 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:25.075785    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj" podUID="a52de307-2a13-4669-8c52-5d53a8d00e93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:03:32.707233 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:32.707205    2555 generic.go:356] "Generic (PLEG): container finished" podID="a52de307-2a13-4669-8c52-5d53a8d00e93" containerID="b66bf921ecc581d946d6f11fc8cdde14dfd347923811822db65a43015d15e04b" exitCode=0
Sep 17 16:03:32.707560 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:32.707282    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj" event={"ID":"a52de307-2a13-4669-8c52-5d53a8d00e93","Type":"ContainerDied","Data":"b66bf921ecc581d946d6f11fc8cdde14dfd347923811822db65a43015d15e04b"}
Sep 17 16:03:32.829792 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:32.829769    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:03:32.953911 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:32.953851    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52de307-2a13-4669-8c52-5d53a8d00e93-tls-certs\") pod \"a52de307-2a13-4669-8c52-5d53a8d00e93\" (UID: \"a52de307-2a13-4669-8c52-5d53a8d00e93\") "
Sep 17 16:03:32.954021 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:32.953916    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9z99\" (UniqueName: \"kubernetes.io/projected/a52de307-2a13-4669-8c52-5d53a8d00e93-kube-api-access-j9z99\") pod \"a52de307-2a13-4669-8c52-5d53a8d00e93\" (UID: \"a52de307-2a13-4669-8c52-5d53a8d00e93\") "
Sep 17 16:03:32.956128 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:32.956099    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52de307-2a13-4669-8c52-5d53a8d00e93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a52de307-2a13-4669-8c52-5d53a8d00e93" (UID: "a52de307-2a13-4669-8c52-5d53a8d00e93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:32.956231 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:32.956141    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52de307-2a13-4669-8c52-5d53a8d00e93-kube-api-access-j9z99" (OuterVolumeSpecName: "kube-api-access-j9z99") pod "a52de307-2a13-4669-8c52-5d53a8d00e93" (UID: "a52de307-2a13-4669-8c52-5d53a8d00e93"). InnerVolumeSpecName "kube-api-access-j9z99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:33.054539 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:33.054516    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52de307-2a13-4669-8c52-5d53a8d00e93-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:03:33.054539 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:33.054537    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9z99\" (UniqueName: \"kubernetes.io/projected/a52de307-2a13-4669-8c52-5d53a8d00e93-kube-api-access-j9z99\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:03:33.713285 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:33.713251    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj" event={"ID":"a52de307-2a13-4669-8c52-5d53a8d00e93","Type":"ContainerDied","Data":"42c4e5aa52c35abf14e9d24460e4240d473d8aa70049be3790f65c79d9221945"}
Sep 17 16:03:33.713714 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:33.713305    2555 scope.go:117] "RemoveContainer" containerID="b66bf921ecc581d946d6f11fc8cdde14dfd347923811822db65a43015d15e04b"
Sep 17 16:03:33.713714 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:33.713260    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"
Sep 17 16:03:33.739986 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:33.739959    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"]
Sep 17 16:03:33.741994 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:33.741973    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5smj"]
Sep 17 16:03:34.700796 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:34.700761    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a52de307-2a13-4669-8c52-5d53a8d00e93" path="/var/lib/kubelet/pods/a52de307-2a13-4669-8c52-5d53a8d00e93/volumes"
Sep 17 16:03:38.757385 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:38.757360    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:03:38.757848 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:38.757740    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:03:51.792461 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.792421    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"]
Sep 17 16:03:51.792905 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.792776    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a52de307-2a13-4669-8c52-5d53a8d00e93" containerName="main"
Sep 17 16:03:51.792905 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.792789    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a52de307-2a13-4669-8c52-5d53a8d00e93" containerName="main"
Sep 17 16:03:51.792905 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.792867    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a52de307-2a13-4669-8c52-5d53a8d00e93" containerName="main"
Sep 17 16:03:51.796625 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.796609    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:03:51.801206 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.801183    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 16:03:51.801206 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.801196    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-j7q5k\""
Sep 17 16:03:51.801424 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.801228    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 16:03:51.801501 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.801484    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 16:03:51.801602 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.801584    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-sdqrv\""
Sep 17 16:03:51.807406 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.807386    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"]
Sep 17 16:03:51.885909 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.885874    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hwtq\" (UniqueName: \"kubernetes.io/projected/f386ec60-d561-4c97-8aba-e23b7b8b73e8-kube-api-access-8hwtq\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57858h94j\" (UID: \"f386ec60-d561-4c97-8aba-e23b7b8b73e8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:03:51.886060 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.885919    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f386ec60-d561-4c97-8aba-e23b7b8b73e8-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57858h94j\" (UID: \"f386ec60-d561-4c97-8aba-e23b7b8b73e8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:03:51.987119 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.987086    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hwtq\" (UniqueName: \"kubernetes.io/projected/f386ec60-d561-4c97-8aba-e23b7b8b73e8-kube-api-access-8hwtq\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57858h94j\" (UID: \"f386ec60-d561-4c97-8aba-e23b7b8b73e8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:03:51.987119 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.987118    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f386ec60-d561-4c97-8aba-e23b7b8b73e8-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57858h94j\" (UID: \"f386ec60-d561-4c97-8aba-e23b7b8b73e8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:03:51.989672 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.989645    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f386ec60-d561-4c97-8aba-e23b7b8b73e8-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57858h94j\" (UID: \"f386ec60-d561-4c97-8aba-e23b7b8b73e8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:03:51.996182 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:51.996163    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hwtq\" (UniqueName: \"kubernetes.io/projected/f386ec60-d561-4c97-8aba-e23b7b8b73e8-kube-api-access-8hwtq\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57858h94j\" (UID: \"f386ec60-d561-4c97-8aba-e23b7b8b73e8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:03:52.108646 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:52.108610    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:03:52.396942 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:52.396841    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"]
Sep 17 16:03:52.399137 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:03:52.399104    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf386ec60_d561_4c97_8aba_e23b7b8b73e8.slice/crio-4ab8a429e222cccc6f02095bf016352ee8084d78acbd8c72c52050847dfde0fa WatchSource:0}: Error finding container 4ab8a429e222cccc6f02095bf016352ee8084d78acbd8c72c52050847dfde0fa: Status 404 returned error can't find the container with id 4ab8a429e222cccc6f02095bf016352ee8084d78acbd8c72c52050847dfde0fa
Sep 17 16:03:52.807848 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:52.807740    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j" event={"ID":"f386ec60-d561-4c97-8aba-e23b7b8b73e8","Type":"ContainerStarted","Data":"e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b"}
Sep 17 16:03:52.807848 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:52.807795    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j" event={"ID":"f386ec60-d561-4c97-8aba-e23b7b8b73e8","Type":"ContainerStarted","Data":"4ab8a429e222cccc6f02095bf016352ee8084d78acbd8c72c52050847dfde0fa"}
Sep 17 16:03:52.808269 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:52.807876    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:03:52.853269 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:03:52.853215    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j" podStartSLOduration=1.853202058 podStartE2EDuration="1.853202058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:03:52.851378018 +0000 UTC m=+1514.742238563" watchObservedRunningTime="2026-09-17 16:03:52.853202058 +0000 UTC m=+1514.744062645"
Sep 17 16:04:23.814712 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:04:23.814683    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:04:43.704233 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:04:43.704200    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"]
Sep 17 16:04:45.654184 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:04:45.654152    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"]
Sep 17 16:04:51.257372 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:04:51.257328    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd" podUID="0849f0db-fd42-446e-aa02-31f1521657b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:04:53.814655 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:04:53.814613    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j" podUID="f386ec60-d561-4c97-8aba-e23b7b8b73e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:04:58.722923 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:04:58.722877    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd" podUID="0849f0db-fd42-446e-aa02-31f1521657b3" containerName="main" containerID="cri-o://8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836" gracePeriod=45
Sep 17 16:05:00.673255 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:00.673196    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j" podUID="f386ec60-d561-4c97-8aba-e23b7b8b73e8" containerName="main" containerID="cri-o://e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b" gracePeriod=45
Sep 17 16:05:01.257442 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:01.257401    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd" podUID="0849f0db-fd42-446e-aa02-31f1521657b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:05:03.814298 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:03.814257    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j" podUID="f386ec60-d561-4c97-8aba-e23b7b8b73e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:05:11.257114 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:11.257075    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd" podUID="0849f0db-fd42-446e-aa02-31f1521657b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:05:11.257744 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:11.257182    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:05:13.813906 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:13.813868    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j" podUID="f386ec60-d561-4c97-8aba-e23b7b8b73e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:05:13.814290 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:13.813966    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:05:21.257004 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:21.256960    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd" podUID="0849f0db-fd42-446e-aa02-31f1521657b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:05:23.814721 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:23.814682    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j" podUID="f386ec60-d561-4c97-8aba-e23b7b8b73e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:05:28.964392 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:28.964366    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:05:29.082139 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.082113    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0849f0db-fd42-446e-aa02-31f1521657b3-tls-certs\") pod \"0849f0db-fd42-446e-aa02-31f1521657b3\" (UID: \"0849f0db-fd42-446e-aa02-31f1521657b3\") "
Sep 17 16:05:29.082284 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.082149    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bstf\" (UniqueName: \"kubernetes.io/projected/0849f0db-fd42-446e-aa02-31f1521657b3-kube-api-access-2bstf\") pod \"0849f0db-fd42-446e-aa02-31f1521657b3\" (UID: \"0849f0db-fd42-446e-aa02-31f1521657b3\") "
Sep 17 16:05:29.084229 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.084207    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0849f0db-fd42-446e-aa02-31f1521657b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0849f0db-fd42-446e-aa02-31f1521657b3" (UID: "0849f0db-fd42-446e-aa02-31f1521657b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:05:29.084333 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.084310    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0849f0db-fd42-446e-aa02-31f1521657b3-kube-api-access-2bstf" (OuterVolumeSpecName: "kube-api-access-2bstf") pod "0849f0db-fd42-446e-aa02-31f1521657b3" (UID: "0849f0db-fd42-446e-aa02-31f1521657b3"). InnerVolumeSpecName "kube-api-access-2bstf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:05:29.183312 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.183291    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0849f0db-fd42-446e-aa02-31f1521657b3-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:05:29.183312 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.183312    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bstf\" (UniqueName: \"kubernetes.io/projected/0849f0db-fd42-446e-aa02-31f1521657b3-kube-api-access-2bstf\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:05:29.277435 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.277405    2555 generic.go:356] "Generic (PLEG): container finished" podID="0849f0db-fd42-446e-aa02-31f1521657b3" containerID="8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836" exitCode=0
Sep 17 16:05:29.277578 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.277450    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"
Sep 17 16:05:29.277578 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.277484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd" event={"ID":"0849f0db-fd42-446e-aa02-31f1521657b3","Type":"ContainerDied","Data":"8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836"}
Sep 17 16:05:29.277578 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.277532    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd" event={"ID":"0849f0db-fd42-446e-aa02-31f1521657b3","Type":"ContainerDied","Data":"e5a0cfc57ef63d395a3a771781c8f395f5563272d3ec003f848adf6b7df9b257"}
Sep 17 16:05:29.277578 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.277553    2555 scope.go:117] "RemoveContainer" containerID="8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836"
Sep 17 16:05:29.288948 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.288929    2555 scope.go:117] "RemoveContainer" containerID="8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836"
Sep 17 16:05:29.289182 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:05:29.289164    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836\": container with ID starting with 8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836 not found: ID does not exist" containerID="8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836"
Sep 17 16:05:29.289239 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.289189    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836"} err="failed to get container status \"8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836\": rpc error: code = NotFound desc = could not find container \"8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836\": container with ID starting with 8a502829b24a04aa52fe33f1b92093d64adee4e6ac73ef60648fcb7658673836 not found: ID does not exist"
Sep 17 16:05:29.304748 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.304724    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"]
Sep 17 16:05:29.306446 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:29.306429    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdsvsd"]
Sep 17 16:05:30.695481 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:30.695455    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0849f0db-fd42-446e-aa02-31f1521657b3" path="/var/lib/kubelet/pods/0849f0db-fd42-446e-aa02-31f1521657b3/volumes"
Sep 17 16:05:30.912536 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:30.912508    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:05:30.996306 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:30.996246    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hwtq\" (UniqueName: \"kubernetes.io/projected/f386ec60-d561-4c97-8aba-e23b7b8b73e8-kube-api-access-8hwtq\") pod \"f386ec60-d561-4c97-8aba-e23b7b8b73e8\" (UID: \"f386ec60-d561-4c97-8aba-e23b7b8b73e8\") "
Sep 17 16:05:30.996306 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:30.996273    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f386ec60-d561-4c97-8aba-e23b7b8b73e8-tls-certs\") pod \"f386ec60-d561-4c97-8aba-e23b7b8b73e8\" (UID: \"f386ec60-d561-4c97-8aba-e23b7b8b73e8\") "
Sep 17 16:05:30.998439 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:30.998414    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f386ec60-d561-4c97-8aba-e23b7b8b73e8-kube-api-access-8hwtq" (OuterVolumeSpecName: "kube-api-access-8hwtq") pod "f386ec60-d561-4c97-8aba-e23b7b8b73e8" (UID: "f386ec60-d561-4c97-8aba-e23b7b8b73e8"). InnerVolumeSpecName "kube-api-access-8hwtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:05:30.998536 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:30.998477    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f386ec60-d561-4c97-8aba-e23b7b8b73e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f386ec60-d561-4c97-8aba-e23b7b8b73e8" (UID: "f386ec60-d561-4c97-8aba-e23b7b8b73e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:05:31.097728 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.097703    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hwtq\" (UniqueName: \"kubernetes.io/projected/f386ec60-d561-4c97-8aba-e23b7b8b73e8-kube-api-access-8hwtq\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:05:31.097728 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.097725    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f386ec60-d561-4c97-8aba-e23b7b8b73e8-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:05:31.290151 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.290079    2555 generic.go:356] "Generic (PLEG): container finished" podID="f386ec60-d561-4c97-8aba-e23b7b8b73e8" containerID="e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b" exitCode=0
Sep 17 16:05:31.290151 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.290127    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"
Sep 17 16:05:31.290294 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.290149    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j" event={"ID":"f386ec60-d561-4c97-8aba-e23b7b8b73e8","Type":"ContainerDied","Data":"e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b"}
Sep 17 16:05:31.290294 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.290194    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j" event={"ID":"f386ec60-d561-4c97-8aba-e23b7b8b73e8","Type":"ContainerDied","Data":"4ab8a429e222cccc6f02095bf016352ee8084d78acbd8c72c52050847dfde0fa"}
Sep 17 16:05:31.290294 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.290217    2555 scope.go:117] "RemoveContainer" containerID="e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b"
Sep 17 16:05:31.299576 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.299556    2555 scope.go:117] "RemoveContainer" containerID="e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b"
Sep 17 16:05:31.299857 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:05:31.299817    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b\": container with ID starting with e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b not found: ID does not exist" containerID="e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b"
Sep 17 16:05:31.299938 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.299868    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b"} err="failed to get container status \"e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b\": rpc error: code = NotFound desc = could not find container \"e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b\": container with ID starting with e4dea340812bc66cd284336412af9748001cd13e49b833b464e4c7d98d5d540b not found: ID does not exist"
Sep 17 16:05:31.314846 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.314800    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"]
Sep 17 16:05:31.320172 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:31.320150    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57858h94j"]
Sep 17 16:05:32.695078 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:05:32.695046    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f386ec60-d561-4c97-8aba-e23b7b8b73e8" path="/var/lib/kubelet/pods/f386ec60-d561-4c97-8aba-e23b7b8b73e8/volumes"
Sep 17 16:06:10.943243 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.943208    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"]
Sep 17 16:06:10.943861 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.943604    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0849f0db-fd42-446e-aa02-31f1521657b3" containerName="main"
Sep 17 16:06:10.943861 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.943621    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0849f0db-fd42-446e-aa02-31f1521657b3" containerName="main"
Sep 17 16:06:10.943861 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.943661    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f386ec60-d561-4c97-8aba-e23b7b8b73e8" containerName="main"
Sep 17 16:06:10.943861 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.943670    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f386ec60-d561-4c97-8aba-e23b7b8b73e8" containerName="main"
Sep 17 16:06:10.943861 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.943743    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0849f0db-fd42-446e-aa02-31f1521657b3" containerName="main"
Sep 17 16:06:10.943861 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.943755    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f386ec60-d561-4c97-8aba-e23b7b8b73e8" containerName="main"
Sep 17 16:06:10.947015 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.946995    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:06:10.952040 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.952017    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 16:06:10.952184 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.952051    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-bkxzz\""
Sep 17 16:06:10.952184 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.952065    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 16:06:10.952184 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.952039    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-hng5x\""
Sep 17 16:06:10.952184 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.952079    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 16:06:10.959283 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:10.959262    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"]
Sep 17 16:06:11.075573 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:11.075544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0869d2-63c1-47c1-b182-b8a288464fbe-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl\" (UID: \"7e0869d2-63c1-47c1-b182-b8a288464fbe\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:06:11.075723 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:11.075591    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cdwq\" (UniqueName: \"kubernetes.io/projected/7e0869d2-63c1-47c1-b182-b8a288464fbe-kube-api-access-4cdwq\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl\" (UID: \"7e0869d2-63c1-47c1-b182-b8a288464fbe\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:06:11.176984 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:11.176949    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0869d2-63c1-47c1-b182-b8a288464fbe-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl\" (UID: \"7e0869d2-63c1-47c1-b182-b8a288464fbe\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:06:11.177156 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:11.177008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cdwq\" (UniqueName: \"kubernetes.io/projected/7e0869d2-63c1-47c1-b182-b8a288464fbe-kube-api-access-4cdwq\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl\" (UID: \"7e0869d2-63c1-47c1-b182-b8a288464fbe\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:06:11.179497 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:11.179473    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0869d2-63c1-47c1-b182-b8a288464fbe-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl\" (UID: \"7e0869d2-63c1-47c1-b182-b8a288464fbe\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:06:11.186559 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:11.186534    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cdwq\" (UniqueName: \"kubernetes.io/projected/7e0869d2-63c1-47c1-b182-b8a288464fbe-kube-api-access-4cdwq\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl\" (UID: \"7e0869d2-63c1-47c1-b182-b8a288464fbe\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:06:11.257046 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:11.256985    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:06:11.542430 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:11.542399    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"]
Sep 17 16:06:11.544300 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:06:11.544268    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0869d2_63c1_47c1_b182_b8a288464fbe.slice/crio-fb8f8a06de818aa3c85bd55544f63e0abe5c57e6f529f31c8fc3d446aff33443 WatchSource:0}: Error finding container fb8f8a06de818aa3c85bd55544f63e0abe5c57e6f529f31c8fc3d446aff33443: Status 404 returned error can't find the container with id fb8f8a06de818aa3c85bd55544f63e0abe5c57e6f529f31c8fc3d446aff33443
Sep 17 16:06:12.490542 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:12.490510    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl" event={"ID":"7e0869d2-63c1-47c1-b182-b8a288464fbe","Type":"ContainerStarted","Data":"a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e"}
Sep 17 16:06:12.490968 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:12.490585    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl" event={"ID":"7e0869d2-63c1-47c1-b182-b8a288464fbe","Type":"ContainerStarted","Data":"fb8f8a06de818aa3c85bd55544f63e0abe5c57e6f529f31c8fc3d446aff33443"}
Sep 17 16:06:12.490968 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:12.490639    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:06:12.513077 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:12.513014    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl" podStartSLOduration=2.513000924 podStartE2EDuration="2.513000924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:12.509502973 +0000 UTC m=+1654.400363516" watchObservedRunningTime="2026-09-17 16:06:12.513000924 +0000 UTC m=+1654.403861484"
Sep 17 16:06:43.498712 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:06:43.498677    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:08:15.517481 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:08:15.517450    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"]
Sep 17 16:08:23.498159 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:08:23.498118    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl" podUID="7e0869d2-63c1-47c1-b182-b8a288464fbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:08:30.535865 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:08:30.535775    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl" podUID="7e0869d2-63c1-47c1-b182-b8a288464fbe" containerName="main" containerID="cri-o://a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e" gracePeriod=45
Sep 17 16:08:33.498139 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:08:33.498099    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl" podUID="7e0869d2-63c1-47c1-b182-b8a288464fbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:08:38.788267 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:08:38.788245    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:08:38.790941 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:08:38.790917    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:08:43.498285 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:08:43.498241    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl" podUID="7e0869d2-63c1-47c1-b182-b8a288464fbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:08:43.498692 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:08:43.498353    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:08:53.498811 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:08:53.498768    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl" podUID="7e0869d2-63c1-47c1-b182-b8a288464fbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:09:00.782269 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:00.782239    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:09:00.868411 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:00.868385    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cdwq\" (UniqueName: \"kubernetes.io/projected/7e0869d2-63c1-47c1-b182-b8a288464fbe-kube-api-access-4cdwq\") pod \"7e0869d2-63c1-47c1-b182-b8a288464fbe\" (UID: \"7e0869d2-63c1-47c1-b182-b8a288464fbe\") "
Sep 17 16:09:00.868529 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:00.868420    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0869d2-63c1-47c1-b182-b8a288464fbe-tls-certs\") pod \"7e0869d2-63c1-47c1-b182-b8a288464fbe\" (UID: \"7e0869d2-63c1-47c1-b182-b8a288464fbe\") "
Sep 17 16:09:00.870521 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:00.870491    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0869d2-63c1-47c1-b182-b8a288464fbe-kube-api-access-4cdwq" (OuterVolumeSpecName: "kube-api-access-4cdwq") pod "7e0869d2-63c1-47c1-b182-b8a288464fbe" (UID: "7e0869d2-63c1-47c1-b182-b8a288464fbe"). InnerVolumeSpecName "kube-api-access-4cdwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:09:00.870627 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:00.870526    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0869d2-63c1-47c1-b182-b8a288464fbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e0869d2-63c1-47c1-b182-b8a288464fbe" (UID: "7e0869d2-63c1-47c1-b182-b8a288464fbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:09:00.969606 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:00.969581    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cdwq\" (UniqueName: \"kubernetes.io/projected/7e0869d2-63c1-47c1-b182-b8a288464fbe-kube-api-access-4cdwq\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:09:00.969606 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:00.969602    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0869d2-63c1-47c1-b182-b8a288464fbe-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:09:01.306361 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:01.306283    2555 generic.go:356] "Generic (PLEG): container finished" podID="7e0869d2-63c1-47c1-b182-b8a288464fbe" containerID="a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e" exitCode=0
Sep 17 16:09:01.306361 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:01.306331    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"
Sep 17 16:09:01.306510 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:01.306359    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl" event={"ID":"7e0869d2-63c1-47c1-b182-b8a288464fbe","Type":"ContainerDied","Data":"a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e"}
Sep 17 16:09:01.306510 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:01.306405    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl" event={"ID":"7e0869d2-63c1-47c1-b182-b8a288464fbe","Type":"ContainerDied","Data":"fb8f8a06de818aa3c85bd55544f63e0abe5c57e6f529f31c8fc3d446aff33443"}
Sep 17 16:09:01.306510 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:01.306428    2555 scope.go:117] "RemoveContainer" containerID="a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e"
Sep 17 16:09:01.315776 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:01.315756    2555 scope.go:117] "RemoveContainer" containerID="a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e"
Sep 17 16:09:01.316083 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:09:01.316063    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e\": container with ID starting with a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e not found: ID does not exist" containerID="a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e"
Sep 17 16:09:01.316162 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:01.316095    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e"} err="failed to get container status \"a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e\": rpc error: code = NotFound desc = could not find container \"a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e\": container with ID starting with a974681464d27ec83ce3dd30fb768a38a9745b7dc0bd817874e700bcddf21c1e not found: ID does not exist"
Sep 17 16:09:01.333775 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:01.333751    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"]
Sep 17 16:09:01.340057 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:01.340035    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8csnl"]
Sep 17 16:09:02.693942 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:02.693906    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e0869d2-63c1-47c1-b182-b8a288464fbe" path="/var/lib/kubelet/pods/7e0869d2-63c1-47c1-b182-b8a288464fbe/volumes"
Sep 17 16:09:05.835188 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.835156    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"]
Sep 17 16:09:05.835531 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.835466    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0869d2-63c1-47c1-b182-b8a288464fbe" containerName="main"
Sep 17 16:09:05.835531 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.835477    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0869d2-63c1-47c1-b182-b8a288464fbe" containerName="main"
Sep 17 16:09:05.835607 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.835547    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0869d2-63c1-47c1-b182-b8a288464fbe" containerName="main"
Sep 17 16:09:05.839671 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.839651    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:05.842437 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.842416    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 16:09:05.842437 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.842428    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 16:09:05.842629 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.842423    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-ftctk\""
Sep 17 16:09:05.842747 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.842733    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 16:09:05.853147 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.853126    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"]
Sep 17 16:09:05.906936 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.906905    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:05.907083 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.906958    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbl4c\" (UniqueName: \"kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-kube-api-access-wbl4c\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:05.907083 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.906990    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:05.907083 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.907017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567b2a74-c980-4167-a271-c3312e31f0b5-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:05.907206 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.907112    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567b2a74-c980-4167-a271-c3312e31f0b5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:05.907206 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.907143    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:05.907206 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.907186    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:05.907324 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.907216    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:05.907324 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:05.907239    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.007968 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.007935    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567b2a74-c980-4167-a271-c3312e31f0b5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.007968 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.007968    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008193 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.007995    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008193 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008017    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008193 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008048    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008193 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008193 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008096    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbl4c\" (UniqueName: \"kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-kube-api-access-wbl4c\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008193 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008124    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008193 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567b2a74-c980-4167-a271-c3312e31f0b5-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008542 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008459    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008688 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008669    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008768 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008698    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008810 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008773    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567b2a74-c980-4167-a271-c3312e31f0b5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.008810 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.008773    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.010968 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.010943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.011090 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.011074    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567b2a74-c980-4167-a271-c3312e31f0b5-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.016918 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.016892    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbl4c\" (UniqueName: \"kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-kube-api-access-wbl4c\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.017321 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.017301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-d8d8q\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.150336 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.150273    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:06.445543 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.445521    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"]
Sep 17 16:09:06.447815 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:09:06.447787    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567b2a74_c980_4167_a271_c3312e31f0b5.slice/crio-2916dd9f309475e57902fbe2c2911c34161b747bd51f8e7dfc1093faa5ff5388 WatchSource:0}: Error finding container 2916dd9f309475e57902fbe2c2911c34161b747bd51f8e7dfc1093faa5ff5388: Status 404 returned error can't find the container with id 2916dd9f309475e57902fbe2c2911c34161b747bd51f8e7dfc1093faa5ff5388
Sep 17 16:09:06.450388 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.450351    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:09:06.450448 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.450427    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:09:06.450487 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:06.450456    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:09:07.339604 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:07.339563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" event={"ID":"567b2a74-c980-4167-a271-c3312e31f0b5","Type":"ContainerStarted","Data":"040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827"}
Sep 17 16:09:07.340040 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:07.339648    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" event={"ID":"567b2a74-c980-4167-a271-c3312e31f0b5","Type":"ContainerStarted","Data":"2916dd9f309475e57902fbe2c2911c34161b747bd51f8e7dfc1093faa5ff5388"}
Sep 17 16:09:07.369840 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:07.369782    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" podStartSLOduration=2.369766883 podStartE2EDuration="2.369766883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:09:07.366777869 +0000 UTC m=+1829.257638430" watchObservedRunningTime="2026-09-17 16:09:07.369766883 +0000 UTC m=+1829.260627427"
Sep 17 16:09:08.151317 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:08.151284    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:08.156417 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:08.156394    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:08.344606 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:08.344580    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:08.345278 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:08.345262    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:14.433615 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.433582    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"]
Sep 17 16:09:14.436643 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.436621    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.442202 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.442180    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-wvkwc\""
Sep 17 16:09:14.443184 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.443161    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 16:09:14.459708 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.459688    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"]
Sep 17 16:09:14.570419 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.570390    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zht\" (UniqueName: \"kubernetes.io/projected/4099e0a7-646a-4276-8e08-cde5e39bc707-kube-api-access-p9zht\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.570519 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.570438    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.570519 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.570456    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4099e0a7-646a-4276-8e08-cde5e39bc707-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.570519 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.570498    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.570519 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.570516    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.570700 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.570541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.671638 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.671605    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.671638 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.671638    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4099e0a7-646a-4276-8e08-cde5e39bc707-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.671894 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.671659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.671894 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.671843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.671894 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.671877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.672079 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.671901    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zht\" (UniqueName: \"kubernetes.io/projected/4099e0a7-646a-4276-8e08-cde5e39bc707-kube-api-access-p9zht\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.672273 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.672241    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.672390 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.672275    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.672390 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.672344    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.673996 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.673968    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.674245 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.674229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4099e0a7-646a-4276-8e08-cde5e39bc707-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.682949 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.682928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zht\" (UniqueName: \"kubernetes.io/projected/4099e0a7-646a-4276-8e08-cde5e39bc707-kube-api-access-p9zht\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:14.746707 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:14.746633    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:15.026591 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:15.026565    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"]
Sep 17 16:09:15.028201 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:09:15.028169    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4099e0a7_646a_4276_8e08_cde5e39bc707.slice/crio-fbfe7f6dcd25d1e7f6e814d86fe6a2f9ac7de173d0050133c9585004b8b5faba WatchSource:0}: Error finding container fbfe7f6dcd25d1e7f6e814d86fe6a2f9ac7de173d0050133c9585004b8b5faba: Status 404 returned error can't find the container with id fbfe7f6dcd25d1e7f6e814d86fe6a2f9ac7de173d0050133c9585004b8b5faba
Sep 17 16:09:15.378174 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:15.378137    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx" event={"ID":"4099e0a7-646a-4276-8e08-cde5e39bc707","Type":"ContainerStarted","Data":"d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883"}
Sep 17 16:09:15.378349 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:15.378196    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx" event={"ID":"4099e0a7-646a-4276-8e08-cde5e39bc707","Type":"ContainerStarted","Data":"fbfe7f6dcd25d1e7f6e814d86fe6a2f9ac7de173d0050133c9585004b8b5faba"}
Sep 17 16:09:15.398627 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:15.398580    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx" podStartSLOduration=1.398564782 podStartE2EDuration="1.398564782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:09:15.397644266 +0000 UTC m=+1837.288504810" watchObservedRunningTime="2026-09-17 16:09:15.398564782 +0000 UTC m=+1837.289425352"
Sep 17 16:09:15.747475 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:15.747377    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:23.585012 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:23.584971    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": read tcp 10.132.0.2:41416->10.132.0.49:15021: read: connection reset by peer"
Sep 17 16:09:24.424795 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:24.424759    2555 generic.go:356] "Generic (PLEG): container finished" podID="567b2a74-c980-4167-a271-c3312e31f0b5" containerID="040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827" exitCode=0
Sep 17 16:09:24.424991 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:24.424843    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" event={"ID":"567b2a74-c980-4167-a271-c3312e31f0b5","Type":"ContainerDied","Data":"040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827"}
Sep 17 16:09:24.425213 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:24.425198    2555 scope.go:117] "RemoveContainer" containerID="040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827"
Sep 17 16:09:24.426135 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:24.426107    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:09:24.426234 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:24.426191    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:09:24.426296 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:24.426267    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:09:24.747168 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:24.747085    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:24.759615 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:24.759591    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:25.431470 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:25.431428    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" event={"ID":"567b2a74-c980-4167-a271-c3312e31f0b5","Type":"ContainerStarted","Data":"89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe"}
Sep 17 16:09:25.450252 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:25.450198    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:09:26.151149 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:26.151113    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:26.151633 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:26.151431    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 16:09:27.150960 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:27.150919    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 16:09:28.151191 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:28.151146    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 16:09:29.151051 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:29.151003    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 16:09:30.151474 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:30.151053    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 16:09:31.154939 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:31.154906    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:31.155391 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:31.155198    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:09:31.155792 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:09:31.155774    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:10:29.591505 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.591461    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"]
Sep 17 16:10:29.594123 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.594102    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.598347 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.598322    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-r8h7d\""
Sep 17 16:10:29.598473 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.598333    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 16:10:29.598473 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.598361    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 16:10:29.598473 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.598444    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 16:10:29.605953 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.605932    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"]
Sep 17 16:10:29.721869 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.721847    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczpb\" (UniqueName: \"kubernetes.io/projected/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-kube-api-access-jczpb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.721976 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.721880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.721976 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.721922    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.721976 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.721955    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.722097 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.721984    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.722097 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.722008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.822643 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.822613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.822796 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.822667    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczpb\" (UniqueName: \"kubernetes.io/projected/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-kube-api-access-jczpb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.822796 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.822689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.822960 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.822934    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.823030 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.822978    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.823088 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.823029    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.823152 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.823137    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.823292 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.823272    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.823382 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.823347    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.825128 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.825102    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.825219 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.825159    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.835284 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.835260    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczpb\" (UniqueName: \"kubernetes.io/projected/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-kube-api-access-jczpb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:29.906042 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:29.905977    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:30.178484 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:30.178455    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"]
Sep 17 16:10:30.179455 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:10:30.179430    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16eed19e_94e6_4e88_b7fe_7c0de2599b5f.slice/crio-b31bc25206f189019bd8ab82e2e92680882d546bbe2dcd16e780871d88879635 WatchSource:0}: Error finding container b31bc25206f189019bd8ab82e2e92680882d546bbe2dcd16e780871d88879635: Status 404 returned error can't find the container with id b31bc25206f189019bd8ab82e2e92680882d546bbe2dcd16e780871d88879635
Sep 17 16:10:30.773045 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:30.773010    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l" event={"ID":"16eed19e-94e6-4e88-b7fe-7c0de2599b5f","Type":"ContainerStarted","Data":"d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489"}
Sep 17 16:10:30.773451 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:30.773064    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l" event={"ID":"16eed19e-94e6-4e88-b7fe-7c0de2599b5f","Type":"ContainerStarted","Data":"b31bc25206f189019bd8ab82e2e92680882d546bbe2dcd16e780871d88879635"}
Sep 17 16:10:30.797205 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:30.797156    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l" podStartSLOduration=1.797140489 podStartE2EDuration="1.797140489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:30.796709872 +0000 UTC m=+1912.687570419" watchObservedRunningTime="2026-09-17 16:10:30.797140489 +0000 UTC m=+1912.688001032"
Sep 17 16:10:30.907091 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:30.907046    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:39.906984 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:39.906944    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:39.919071 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:39.919046    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:40.832204 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:40.832169    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:10:44.185858 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.185802    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"]
Sep 17 16:10:44.194693 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:10:44.194646    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:10:44.194693 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:10:44.194693 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:10:44.194693 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883" cmd=["/bin/sleep","15"]
Sep 17 16:10:44.194972 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.194708    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx" message="" err=<
Sep 17 16:10:44.194972 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:10:44.194972 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:10:44.194972 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 16:10:44.194972 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:10:44.194721    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:10:44.194972 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:10:44.194972 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:10:44.194972 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx" podUID="4099e0a7-646a-4276-8e08-cde5e39bc707" containerName="main" containerID="cri-o://d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883"
Sep 17 16:10:44.194972 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.194759    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx" podUID="4099e0a7-646a-4276-8e08-cde5e39bc707" containerName="main" containerID="cri-o://d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883" gracePeriod=60
Sep 17 16:10:44.432038 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.432004    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx" podUID="4099e0a7-646a-4276-8e08-cde5e39bc707" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 17 16:10:44.529923 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.529903    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:10:44.637054 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637026    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-model-cache\") pod \"4099e0a7-646a-4276-8e08-cde5e39bc707\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") "
Sep 17 16:10:44.637182 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637084    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-dshm\") pod \"4099e0a7-646a-4276-8e08-cde5e39bc707\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") "
Sep 17 16:10:44.637182 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637124    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4099e0a7-646a-4276-8e08-cde5e39bc707-tls-certs\") pod \"4099e0a7-646a-4276-8e08-cde5e39bc707\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") "
Sep 17 16:10:44.637182 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637153    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-home\") pod \"4099e0a7-646a-4276-8e08-cde5e39bc707\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") "
Sep 17 16:10:44.637346 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637183    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-tmp-dir\") pod \"4099e0a7-646a-4276-8e08-cde5e39bc707\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") "
Sep 17 16:10:44.637346 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637232    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9zht\" (UniqueName: \"kubernetes.io/projected/4099e0a7-646a-4276-8e08-cde5e39bc707-kube-api-access-p9zht\") pod \"4099e0a7-646a-4276-8e08-cde5e39bc707\" (UID: \"4099e0a7-646a-4276-8e08-cde5e39bc707\") "
Sep 17 16:10:44.637346 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637264    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-model-cache" (OuterVolumeSpecName: "model-cache") pod "4099e0a7-646a-4276-8e08-cde5e39bc707" (UID: "4099e0a7-646a-4276-8e08-cde5e39bc707"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:44.637485 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637400    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-home" (OuterVolumeSpecName: "home") pod "4099e0a7-646a-4276-8e08-cde5e39bc707" (UID: "4099e0a7-646a-4276-8e08-cde5e39bc707"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:44.637485 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637435    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4099e0a7-646a-4276-8e08-cde5e39bc707" (UID: "4099e0a7-646a-4276-8e08-cde5e39bc707"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:44.637606 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637502    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:44.637606 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637525    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:44.637606 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.637542    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:44.639344 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.639323    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4099e0a7-646a-4276-8e08-cde5e39bc707-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4099e0a7-646a-4276-8e08-cde5e39bc707" (UID: "4099e0a7-646a-4276-8e08-cde5e39bc707"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:10:44.639344 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.639325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-dshm" (OuterVolumeSpecName: "dshm") pod "4099e0a7-646a-4276-8e08-cde5e39bc707" (UID: "4099e0a7-646a-4276-8e08-cde5e39bc707"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:44.639484 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.639386    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4099e0a7-646a-4276-8e08-cde5e39bc707-kube-api-access-p9zht" (OuterVolumeSpecName: "kube-api-access-p9zht") pod "4099e0a7-646a-4276-8e08-cde5e39bc707" (UID: "4099e0a7-646a-4276-8e08-cde5e39bc707"). InnerVolumeSpecName "kube-api-access-p9zht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:10:44.738062 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.738000    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9zht\" (UniqueName: \"kubernetes.io/projected/4099e0a7-646a-4276-8e08-cde5e39bc707-kube-api-access-p9zht\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:44.738062 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.738032    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4099e0a7-646a-4276-8e08-cde5e39bc707-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:44.738062 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.738042    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4099e0a7-646a-4276-8e08-cde5e39bc707-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:44.845585 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.845546    2555 generic.go:356] "Generic (PLEG): container finished" podID="4099e0a7-646a-4276-8e08-cde5e39bc707" containerID="d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883" exitCode=0
Sep 17 16:10:44.845704 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.845609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx" event={"ID":"4099e0a7-646a-4276-8e08-cde5e39bc707","Type":"ContainerDied","Data":"d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883"}
Sep 17 16:10:44.845704 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.845617    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"
Sep 17 16:10:44.845704 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.845677    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx" event={"ID":"4099e0a7-646a-4276-8e08-cde5e39bc707","Type":"ContainerDied","Data":"fbfe7f6dcd25d1e7f6e814d86fe6a2f9ac7de173d0050133c9585004b8b5faba"}
Sep 17 16:10:44.845704 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.845700    2555 scope.go:117] "RemoveContainer" containerID="d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883"
Sep 17 16:10:44.854505 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.854490    2555 scope.go:117] "RemoveContainer" containerID="d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883"
Sep 17 16:10:44.854849 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:10:44.854809    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883\": container with ID starting with d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883 not found: ID does not exist" containerID="d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883"
Sep 17 16:10:44.854920 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.854858    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883"} err="failed to get container status \"d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883\": rpc error: code = NotFound desc = could not find container \"d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883\": container with ID starting with d36b0f376dcac431d4bb0c14ba8da9b9072a15aab82cd2c363835287c94aa883 not found: ID does not exist"
Sep 17 16:10:44.871771 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.871746    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"]
Sep 17 16:10:44.879174 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:44.879155    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997p5wdx"]
Sep 17 16:10:46.695411 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:46.695369    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4099e0a7-646a-4276-8e08-cde5e39bc707" path="/var/lib/kubelet/pods/4099e0a7-646a-4276-8e08-cde5e39bc707/volumes"
Sep 17 16:10:54.375029 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:54.374998    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"]
Sep 17 16:10:54.375461 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:54.375207    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy" containerID="cri-o://89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe" gracePeriod=30
Sep 17 16:10:59.616183 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.616161    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:10:59.752616 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752589    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-data\") pod \"567b2a74-c980-4167-a271-c3312e31f0b5\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") "
Sep 17 16:10:59.752754 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752655    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567b2a74-c980-4167-a271-c3312e31f0b5-istio-podinfo\") pod \"567b2a74-c980-4167-a271-c3312e31f0b5\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") "
Sep 17 16:10:59.752754 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752681    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-credential-socket\") pod \"567b2a74-c980-4167-a271-c3312e31f0b5\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") "
Sep 17 16:10:59.752754 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752718    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567b2a74-c980-4167-a271-c3312e31f0b5-istiod-ca-cert\") pod \"567b2a74-c980-4167-a271-c3312e31f0b5\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") "
Sep 17 16:10:59.752754 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752743    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-socket\") pod \"567b2a74-c980-4167-a271-c3312e31f0b5\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") "
Sep 17 16:10:59.753007 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752785    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-istio-token\") pod \"567b2a74-c980-4167-a271-c3312e31f0b5\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") "
Sep 17 16:10:59.753007 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752814    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-envoy\") pod \"567b2a74-c980-4167-a271-c3312e31f0b5\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") "
Sep 17 16:10:59.753007 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752871    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbl4c\" (UniqueName: \"kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-kube-api-access-wbl4c\") pod \"567b2a74-c980-4167-a271-c3312e31f0b5\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") "
Sep 17 16:10:59.753007 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752924    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-certs\") pod \"567b2a74-c980-4167-a271-c3312e31f0b5\" (UID: \"567b2a74-c980-4167-a271-c3312e31f0b5\") "
Sep 17 16:10:59.753007 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752864    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "567b2a74-c980-4167-a271-c3312e31f0b5" (UID: "567b2a74-c980-4167-a271-c3312e31f0b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:59.753007 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.752948    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "567b2a74-c980-4167-a271-c3312e31f0b5" (UID: "567b2a74-c980-4167-a271-c3312e31f0b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:59.753283 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.753142    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-data\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:59.753283 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.753158    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-credential-socket\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:59.753283 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.753155    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "567b2a74-c980-4167-a271-c3312e31f0b5" (UID: "567b2a74-c980-4167-a271-c3312e31f0b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:59.753391 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.753286    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567b2a74-c980-4167-a271-c3312e31f0b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "567b2a74-c980-4167-a271-c3312e31f0b5" (UID: "567b2a74-c980-4167-a271-c3312e31f0b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:10:59.753391 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.753349    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "567b2a74-c980-4167-a271-c3312e31f0b5" (UID: "567b2a74-c980-4167-a271-c3312e31f0b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:59.754938 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.754905    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "567b2a74-c980-4167-a271-c3312e31f0b5" (UID: "567b2a74-c980-4167-a271-c3312e31f0b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:10:59.755047 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.755031    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "567b2a74-c980-4167-a271-c3312e31f0b5" (UID: "567b2a74-c980-4167-a271-c3312e31f0b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:59.755260 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.755238    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/567b2a74-c980-4167-a271-c3312e31f0b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "567b2a74-c980-4167-a271-c3312e31f0b5" (UID: "567b2a74-c980-4167-a271-c3312e31f0b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 16:10:59.755358 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.755340    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-kube-api-access-wbl4c" (OuterVolumeSpecName: "kube-api-access-wbl4c") pod "567b2a74-c980-4167-a271-c3312e31f0b5" (UID: "567b2a74-c980-4167-a271-c3312e31f0b5"). InnerVolumeSpecName "kube-api-access-wbl4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:10:59.853751 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.853726    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567b2a74-c980-4167-a271-c3312e31f0b5-istio-podinfo\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:59.853751 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.853745    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567b2a74-c980-4167-a271-c3312e31f0b5-istiod-ca-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:59.853751 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.853755    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-socket\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:59.853981 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.853764    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-istio-token\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:59.853981 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.853778    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-istio-envoy\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:59.853981 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.853790    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbl4c\" (UniqueName: \"kubernetes.io/projected/567b2a74-c980-4167-a271-c3312e31f0b5-kube-api-access-wbl4c\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:59.853981 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.853798    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567b2a74-c980-4167-a271-c3312e31f0b5-workload-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:10:59.919829 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.919801    2555 generic.go:356] "Generic (PLEG): container finished" podID="567b2a74-c980-4167-a271-c3312e31f0b5" containerID="89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe" exitCode=0
Sep 17 16:10:59.919946 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.919853    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"
Sep 17 16:10:59.919946 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.919850    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" event={"ID":"567b2a74-c980-4167-a271-c3312e31f0b5","Type":"ContainerDied","Data":"89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe"}
Sep 17 16:10:59.920021 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.919971    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q" event={"ID":"567b2a74-c980-4167-a271-c3312e31f0b5","Type":"ContainerDied","Data":"2916dd9f309475e57902fbe2c2911c34161b747bd51f8e7dfc1093faa5ff5388"}
Sep 17 16:10:59.920021 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.919992    2555 scope.go:117] "RemoveContainer" containerID="89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe"
Sep 17 16:10:59.928794 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.928773    2555 scope.go:117] "RemoveContainer" containerID="040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827"
Sep 17 16:10:59.936447 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.936430    2555 scope.go:117] "RemoveContainer" containerID="89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe"
Sep 17 16:10:59.936710 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:10:59.936692    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe\": container with ID starting with 89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe not found: ID does not exist" containerID="89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe"
Sep 17 16:10:59.936754 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.936719    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe"} err="failed to get container status \"89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe\": rpc error: code = NotFound desc = could not find container \"89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe\": container with ID starting with 89d03e4e63fa2b903385f1705a5b82de737944f2e11a1693a18d6ef49db100fe not found: ID does not exist"
Sep 17 16:10:59.936754 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.936735    2555 scope.go:117] "RemoveContainer" containerID="040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827"
Sep 17 16:10:59.936983 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:10:59.936966    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827\": container with ID starting with 040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827 not found: ID does not exist" containerID="040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827"
Sep 17 16:10:59.937030 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.936988    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827"} err="failed to get container status \"040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827\": rpc error: code = NotFound desc = could not find container \"040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827\": container with ID starting with 040f7df5bd55d1e383415f7ec0a1ff6609a8a388b5519733132c39f44fc6b827 not found: ID does not exist"
Sep 17 16:10:59.943016 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.942996    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"]
Sep 17 16:10:59.948013 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:10:59.947993    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-d8d8q"]
Sep 17 16:11:00.695027 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:00.694999    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" path="/var/lib/kubelet/pods/567b2a74-c980-4167-a271-c3312e31f0b5/volumes"
Sep 17 16:11:07.963425 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.963390    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"]
Sep 17 16:11:07.963806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.963668    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy"
Sep 17 16:11:07.963806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.963679    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy"
Sep 17 16:11:07.963806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.963690    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy"
Sep 17 16:11:07.963806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.963695    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy"
Sep 17 16:11:07.963806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.963708    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4099e0a7-646a-4276-8e08-cde5e39bc707" containerName="main"
Sep 17 16:11:07.963806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.963713    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4099e0a7-646a-4276-8e08-cde5e39bc707" containerName="main"
Sep 17 16:11:07.963806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.963761    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy"
Sep 17 16:11:07.963806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.963769    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="567b2a74-c980-4167-a271-c3312e31f0b5" containerName="istio-proxy"
Sep 17 16:11:07.963806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.963774    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4099e0a7-646a-4276-8e08-cde5e39bc707" containerName="main"
Sep 17 16:11:07.967891 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.967872    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:11:07.971882 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.971854    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 16:11:07.971882 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.971859    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 16:11:07.972116 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.971861    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-p8fv4\""
Sep 17 16:11:07.972116 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.971909    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-w576q\""
Sep 17 16:11:07.972116 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.971948    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 16:11:07.977297 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:07.977276    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"]
Sep 17 16:11:08.008278 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.008259    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thkg\" (UniqueName: \"kubernetes.io/projected/77590019-19d5-4a85-b20d-2822650d4483-kube-api-access-9thkg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2\" (UID: \"77590019-19d5-4a85-b20d-2822650d4483\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:11:08.008373 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.008289    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77590019-19d5-4a85-b20d-2822650d4483-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2\" (UID: \"77590019-19d5-4a85-b20d-2822650d4483\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:11:08.108897 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.108880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thkg\" (UniqueName: \"kubernetes.io/projected/77590019-19d5-4a85-b20d-2822650d4483-kube-api-access-9thkg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2\" (UID: \"77590019-19d5-4a85-b20d-2822650d4483\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:11:08.109009 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.108911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77590019-19d5-4a85-b20d-2822650d4483-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2\" (UID: \"77590019-19d5-4a85-b20d-2822650d4483\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:11:08.111102 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.111085    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77590019-19d5-4a85-b20d-2822650d4483-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2\" (UID: \"77590019-19d5-4a85-b20d-2822650d4483\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:11:08.118441 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.118424    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thkg\" (UniqueName: \"kubernetes.io/projected/77590019-19d5-4a85-b20d-2822650d4483-kube-api-access-9thkg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2\" (UID: \"77590019-19d5-4a85-b20d-2822650d4483\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:11:08.279107 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.279034    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:11:08.554766 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.554665    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"]
Sep 17 16:11:08.557670 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:11:08.557632    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77590019_19d5_4a85_b20d_2822650d4483.slice/crio-96aed59c392d6ce53a956b877063281a0d16afddb46ff298716e78827a1861f8 WatchSource:0}: Error finding container 96aed59c392d6ce53a956b877063281a0d16afddb46ff298716e78827a1861f8: Status 404 returned error can't find the container with id 96aed59c392d6ce53a956b877063281a0d16afddb46ff298716e78827a1861f8
Sep 17 16:11:08.963390 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.963359    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2" event={"ID":"77590019-19d5-4a85-b20d-2822650d4483","Type":"ContainerStarted","Data":"b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2"}
Sep 17 16:11:08.963573 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.963409    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2" event={"ID":"77590019-19d5-4a85-b20d-2822650d4483","Type":"ContainerStarted","Data":"96aed59c392d6ce53a956b877063281a0d16afddb46ff298716e78827a1861f8"}
Sep 17 16:11:08.963573 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.963474    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:11:08.985764 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:08.985717    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2" podStartSLOduration=1.985699983 podStartE2EDuration="1.985699983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:11:08.985683587 +0000 UTC m=+1950.876544131" watchObservedRunningTime="2026-09-17 16:11:08.985699983 +0000 UTC m=+1950.876560531"
Sep 17 16:11:21.954103 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:21.954070    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"]
Sep 17 16:11:21.962956 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:11:21.962918    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:11:21.962956 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:11:21.962956 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:11:21.962956 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489" cmd=["/bin/sleep","15"]
Sep 17 16:11:21.963276 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:21.962972    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l" message="" err=<
Sep 17 16:11:21.963276 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:11:21.963276 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:11:21.963276 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 16:11:21.963276 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:11:21.962986    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:11:21.963276 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:11:21.963276 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:11:21.963276 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l" podUID="16eed19e-94e6-4e88-b7fe-7c0de2599b5f" containerName="main" containerID="cri-o://d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489"
Sep 17 16:11:21.963276 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:21.963042    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l" podUID="16eed19e-94e6-4e88-b7fe-7c0de2599b5f" containerName="main" containerID="cri-o://d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489" gracePeriod=60
Sep 17 16:11:22.315999 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.315974    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:11:22.406934 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.406909    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-model-cache\") pod \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") "
Sep 17 16:11:22.407076 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.406942    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-home\") pod \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") "
Sep 17 16:11:22.407076 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.406961    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-dshm\") pod \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") "
Sep 17 16:11:22.407076 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.406976    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tls-certs\") pod \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") "
Sep 17 16:11:22.407076 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.407026    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tmp-dir\") pod \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") "
Sep 17 16:11:22.407076 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.407053    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jczpb\" (UniqueName: \"kubernetes.io/projected/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-kube-api-access-jczpb\") pod \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\" (UID: \"16eed19e-94e6-4e88-b7fe-7c0de2599b5f\") "
Sep 17 16:11:22.407363 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.407180    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-home" (OuterVolumeSpecName: "home") pod "16eed19e-94e6-4e88-b7fe-7c0de2599b5f" (UID: "16eed19e-94e6-4e88-b7fe-7c0de2599b5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:11:22.407363 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.407196    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "16eed19e-94e6-4e88-b7fe-7c0de2599b5f" (UID: "16eed19e-94e6-4e88-b7fe-7c0de2599b5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:11:22.407363 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.407257    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:11:22.407363 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.407275    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:11:22.407363 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.407330    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16eed19e-94e6-4e88-b7fe-7c0de2599b5f" (UID: "16eed19e-94e6-4e88-b7fe-7c0de2599b5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:11:22.409229 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.409198    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-dshm" (OuterVolumeSpecName: "dshm") pod "16eed19e-94e6-4e88-b7fe-7c0de2599b5f" (UID: "16eed19e-94e6-4e88-b7fe-7c0de2599b5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:11:22.409335 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.409229    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-kube-api-access-jczpb" (OuterVolumeSpecName: "kube-api-access-jczpb") pod "16eed19e-94e6-4e88-b7fe-7c0de2599b5f" (UID: "16eed19e-94e6-4e88-b7fe-7c0de2599b5f"). InnerVolumeSpecName "kube-api-access-jczpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:11:22.409335 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.409259    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16eed19e-94e6-4e88-b7fe-7c0de2599b5f" (UID: "16eed19e-94e6-4e88-b7fe-7c0de2599b5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:11:22.507958 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.507896    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:11:22.507958 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.507921    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jczpb\" (UniqueName: \"kubernetes.io/projected/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-kube-api-access-jczpb\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:11:22.507958 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.507931    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:11:22.507958 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:22.507939    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16eed19e-94e6-4e88-b7fe-7c0de2599b5f-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:11:23.031504 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:23.031453    2555 generic.go:356] "Generic (PLEG): container finished" podID="16eed19e-94e6-4e88-b7fe-7c0de2599b5f" containerID="d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489" exitCode=0
Sep 17 16:11:23.031908 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:23.031513    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"
Sep 17 16:11:23.031908 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:23.031538    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l" event={"ID":"16eed19e-94e6-4e88-b7fe-7c0de2599b5f","Type":"ContainerDied","Data":"d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489"}
Sep 17 16:11:23.031908 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:23.031594    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l" event={"ID":"16eed19e-94e6-4e88-b7fe-7c0de2599b5f","Type":"ContainerDied","Data":"b31bc25206f189019bd8ab82e2e92680882d546bbe2dcd16e780871d88879635"}
Sep 17 16:11:23.031908 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:23.031621    2555 scope.go:117] "RemoveContainer" containerID="d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489"
Sep 17 16:11:23.040402 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:23.040382    2555 scope.go:117] "RemoveContainer" containerID="d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489"
Sep 17 16:11:23.040705 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:11:23.040684    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489\": container with ID starting with d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489 not found: ID does not exist" containerID="d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489"
Sep 17 16:11:23.040762 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:23.040714    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489"} err="failed to get container status \"d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489\": rpc error: code = NotFound desc = could not find container \"d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489\": container with ID starting with d4efb1ecb2c133c86a32e89ed0e3e09acd4a1f896c8ab614b516aeef64cce489 not found: ID does not exist"
Sep 17 16:11:23.054385 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:23.054365    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"]
Sep 17 16:11:23.061696 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:23.061677    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n527l"]
Sep 17 16:11:24.693982 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:24.693950    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16eed19e-94e6-4e88-b7fe-7c0de2599b5f" path="/var/lib/kubelet/pods/16eed19e-94e6-4e88-b7fe-7c0de2599b5f/volumes"
Sep 17 16:11:39.970641 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:11:39.970609    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:13:23.285721 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:23.285686    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"]
Sep 17 16:13:29.970929 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:29.970893    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2" podUID="77590019-19d5-4a85-b20d-2822650d4483" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:13:38.305091 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:38.305022    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2" podUID="77590019-19d5-4a85-b20d-2822650d4483" containerName="main" containerID="cri-o://b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2" gracePeriod=45
Sep 17 16:13:38.822903 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:38.822875    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:13:38.825547 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:38.825515    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:13:39.970057 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:39.970024    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2" podUID="77590019-19d5-4a85-b20d-2822650d4483" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:13:49.970914 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:49.970876    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2" podUID="77590019-19d5-4a85-b20d-2822650d4483" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:13:49.971376 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:49.970990    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:13:52.094523 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.094491    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"]
Sep 17 16:13:52.094968 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.094790    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16eed19e-94e6-4e88-b7fe-7c0de2599b5f" containerName="main"
Sep 17 16:13:52.094968 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.094802    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="16eed19e-94e6-4e88-b7fe-7c0de2599b5f" containerName="main"
Sep 17 16:13:52.094968 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.094885    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="16eed19e-94e6-4e88-b7fe-7c0de2599b5f" containerName="main"
Sep 17 16:13:52.097657 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.097638    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.107320 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.107300    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 16:13:52.107423 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.107371    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 16:13:52.107423 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.107384    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 16:13:52.112271 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.112257    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-kkjx7\""
Sep 17 16:13:52.134447 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.134419    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"]
Sep 17 16:13:52.158681 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.158661    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.158781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.158707    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.158781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.158740    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.158781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.158762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.158918 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.158802    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18398b06-3eed-4089-a429-61973020603b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.158918 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.158847    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qh5z\" (UniqueName: \"kubernetes.io/projected/18398b06-3eed-4089-a429-61973020603b-kube-api-access-5qh5z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.259378 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.259353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.259472 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.259396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.259472 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.259419    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.259585 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.259550    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.259639 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.259610    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18398b06-3eed-4089-a429-61973020603b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.259694 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.259656    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qh5z\" (UniqueName: \"kubernetes.io/projected/18398b06-3eed-4089-a429-61973020603b-kube-api-access-5qh5z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.259753 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.259738    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.259805 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.259772    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.259933 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.259915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.261662 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.261643    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.262027 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.262010    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18398b06-3eed-4089-a429-61973020603b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.269629 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.269609    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qh5z\" (UniqueName: \"kubernetes.io/projected/18398b06-3eed-4089-a429-61973020603b-kube-api-access-5qh5z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.408040 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.407974    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:52.688308 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.688284    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"]
Sep 17 16:13:52.690635 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:13:52.690605    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18398b06_3eed_4089_a429_61973020603b.slice/crio-ed29b009180ed94cd0fc6419b503a16aaaf4141fb4acfb05ca39c2c6ac3423a3 WatchSource:0}: Error finding container ed29b009180ed94cd0fc6419b503a16aaaf4141fb4acfb05ca39c2c6ac3423a3: Status 404 returned error can't find the container with id ed29b009180ed94cd0fc6419b503a16aaaf4141fb4acfb05ca39c2c6ac3423a3
Sep 17 16:13:52.746187 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:52.746158    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w" event={"ID":"18398b06-3eed-4089-a429-61973020603b","Type":"ContainerStarted","Data":"ed29b009180ed94cd0fc6419b503a16aaaf4141fb4acfb05ca39c2c6ac3423a3"}
Sep 17 16:13:53.752687 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:53.752652    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w" event={"ID":"18398b06-3eed-4089-a429-61973020603b","Type":"ContainerStarted","Data":"daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22"}
Sep 17 16:13:53.779752 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:53.779706    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w" podStartSLOduration=1.779690166 podStartE2EDuration="1.779690166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:13:53.778082883 +0000 UTC m=+2115.668943440" watchObservedRunningTime="2026-09-17 16:13:53.779690166 +0000 UTC m=+2115.670550710"
Sep 17 16:13:54.408470 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:54.408436    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:13:59.971932 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:13:59.971895    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2" podUID="77590019-19d5-4a85-b20d-2822650d4483" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:14:02.408988 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:02.408954    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:14:02.421160 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:02.421135    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:14:02.805234 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:02.805154    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:14:08.543765 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.543743    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:14:08.678974 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.678946    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77590019-19d5-4a85-b20d-2822650d4483-tls-certs\") pod \"77590019-19d5-4a85-b20d-2822650d4483\" (UID: \"77590019-19d5-4a85-b20d-2822650d4483\") "
Sep 17 16:14:08.679110 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.679001    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9thkg\" (UniqueName: \"kubernetes.io/projected/77590019-19d5-4a85-b20d-2822650d4483-kube-api-access-9thkg\") pod \"77590019-19d5-4a85-b20d-2822650d4483\" (UID: \"77590019-19d5-4a85-b20d-2822650d4483\") "
Sep 17 16:14:08.681229 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.681200    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77590019-19d5-4a85-b20d-2822650d4483-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77590019-19d5-4a85-b20d-2822650d4483" (UID: "77590019-19d5-4a85-b20d-2822650d4483"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:14:08.681229 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.681204    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77590019-19d5-4a85-b20d-2822650d4483-kube-api-access-9thkg" (OuterVolumeSpecName: "kube-api-access-9thkg") pod "77590019-19d5-4a85-b20d-2822650d4483" (UID: "77590019-19d5-4a85-b20d-2822650d4483"). InnerVolumeSpecName "kube-api-access-9thkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:14:08.779794 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.779770    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9thkg\" (UniqueName: \"kubernetes.io/projected/77590019-19d5-4a85-b20d-2822650d4483-kube-api-access-9thkg\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:14:08.779794 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.779792    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77590019-19d5-4a85-b20d-2822650d4483-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:14:08.822885 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.822857    2555 generic.go:356] "Generic (PLEG): container finished" podID="77590019-19d5-4a85-b20d-2822650d4483" containerID="b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2" exitCode=0
Sep 17 16:14:08.823003 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.822897    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"
Sep 17 16:14:08.823003 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.822937    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2" event={"ID":"77590019-19d5-4a85-b20d-2822650d4483","Type":"ContainerDied","Data":"b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2"}
Sep 17 16:14:08.823003 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.822986    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2" event={"ID":"77590019-19d5-4a85-b20d-2822650d4483","Type":"ContainerDied","Data":"96aed59c392d6ce53a956b877063281a0d16afddb46ff298716e78827a1861f8"}
Sep 17 16:14:08.823154 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.823007    2555 scope.go:117] "RemoveContainer" containerID="b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2"
Sep 17 16:14:08.831198 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.831181    2555 scope.go:117] "RemoveContainer" containerID="b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2"
Sep 17 16:14:08.831505 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:14:08.831477    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2\": container with ID starting with b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2 not found: ID does not exist" containerID="b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2"
Sep 17 16:14:08.831505 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.831501    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2"} err="failed to get container status \"b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2\": rpc error: code = NotFound desc = could not find container \"b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2\": container with ID starting with b257b70d21353cceb72e40975c5fb9c49396481fcffb404425777d57eb0558d2 not found: ID does not exist"
Sep 17 16:14:08.841662 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.841639    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"]
Sep 17 16:14:08.845543 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:08.845522    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schekjff2"]
Sep 17 16:14:10.694584 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:14:10.694552    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77590019-19d5-4a85-b20d-2822650d4483" path="/var/lib/kubelet/pods/77590019-19d5-4a85-b20d-2822650d4483/volumes"
Sep 17 16:15:00.186656 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.186621    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"]
Sep 17 16:15:00.187198 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.187082    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77590019-19d5-4a85-b20d-2822650d4483" containerName="main"
Sep 17 16:15:00.187198 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.187101    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="77590019-19d5-4a85-b20d-2822650d4483" containerName="main"
Sep 17 16:15:00.187198 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.187184    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="77590019-19d5-4a85-b20d-2822650d4483" containerName="main"
Sep 17 16:15:00.190331 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.190306    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.193120 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.193097    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 16:15:00.193218 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.193164    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 16:15:00.193447 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.193430    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 16:15:00.193493 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.193435    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-hjx9r\""
Sep 17 16:15:00.204287 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.204263    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"]
Sep 17 16:15:00.350328 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.350292    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.350328 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.350329    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.350597 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.350351    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42zjk\" (UniqueName: \"kubernetes.io/projected/f1637746-ba8c-494c-adca-67c118ddabae-kube-api-access-42zjk\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.350597 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.350464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.350597 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.350526    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.350597 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.350566    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1637746-ba8c-494c-adca-67c118ddabae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.350770 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.350598    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.451815 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.451722    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.451815 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.451769    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.451815 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.451797    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42zjk\" (UniqueName: \"kubernetes.io/projected/f1637746-ba8c-494c-adca-67c118ddabae-kube-api-access-42zjk\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.452118 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.451867    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.452118 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.451918    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.452118 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.451954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1637746-ba8c-494c-adca-67c118ddabae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.452118 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.452068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.452342 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.452149    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.452342 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.452184    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.452342 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.452285    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.452483 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.452389    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.454128 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.454106    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.454535 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.454511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1637746-ba8c-494c-adca-67c118ddabae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.461361 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.461336    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42zjk\" (UniqueName: \"kubernetes.io/projected/f1637746-ba8c-494c-adca-67c118ddabae-kube-api-access-42zjk\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.500320 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.500290    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:00.777886 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.777863    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"]
Sep 17 16:15:00.818629 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:00.818609    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:15:01.075595 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:01.075562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" event={"ID":"f1637746-ba8c-494c-adca-67c118ddabae","Type":"ContainerStarted","Data":"4928a2129086ad3db0c35ecc296f15bedb9c87ff7080e456845488a3eede44cf"}
Sep 17 16:15:09.117072 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:09.116985    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" event={"ID":"f1637746-ba8c-494c-adca-67c118ddabae","Type":"ContainerStarted","Data":"207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8"}
Sep 17 16:15:10.123497 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:10.123459    2555 generic.go:356] "Generic (PLEG): container finished" podID="f1637746-ba8c-494c-adca-67c118ddabae" containerID="207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8" exitCode=0
Sep 17 16:15:10.123942 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:10.123520    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" event={"ID":"f1637746-ba8c-494c-adca-67c118ddabae","Type":"ContainerDied","Data":"207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8"}
Sep 17 16:15:37.025081 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:37.025040    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"]
Sep 17 16:15:37.035162 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:15:37.035112    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:15:37.035162 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:15:37.035162 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:15:37.035162 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22" cmd=["/bin/sleep","15"]
Sep 17 16:15:37.035490 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:37.035190    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w" message="" err=<
Sep 17 16:15:37.035490 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:15:37.035490 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:15:37.035490 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 16:15:37.035490 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:15:37.035213    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:15:37.035490 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:15:37.035490 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:15:37.035490 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w" podUID="18398b06-3eed-4089-a429-61973020603b" containerName="main" containerID="cri-o://daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22"
Sep 17 16:15:37.035490 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:37.035264    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w" podUID="18398b06-3eed-4089-a429-61973020603b" containerName="main" containerID="cri-o://daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22" gracePeriod=60
Sep 17 16:15:37.794735 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:37.794692    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w" podUID="18398b06-3eed-4089-a429-61973020603b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 16:15:38.568058 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.568018    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:15:38.686132 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.686015    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qh5z\" (UniqueName: \"kubernetes.io/projected/18398b06-3eed-4089-a429-61973020603b-kube-api-access-5qh5z\") pod \"18398b06-3eed-4089-a429-61973020603b\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") "
Sep 17 16:15:38.686132 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.686098    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-model-cache\") pod \"18398b06-3eed-4089-a429-61973020603b\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") "
Sep 17 16:15:38.686394 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.686141    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-dshm\") pod \"18398b06-3eed-4089-a429-61973020603b\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") "
Sep 17 16:15:38.686394 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.686184    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18398b06-3eed-4089-a429-61973020603b-tls-certs\") pod \"18398b06-3eed-4089-a429-61973020603b\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") "
Sep 17 16:15:38.686394 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.686217    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-home\") pod \"18398b06-3eed-4089-a429-61973020603b\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") "
Sep 17 16:15:38.686394 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.686232    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-tmp-dir\") pod \"18398b06-3eed-4089-a429-61973020603b\" (UID: \"18398b06-3eed-4089-a429-61973020603b\") "
Sep 17 16:15:38.686599 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.686417    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-model-cache" (OuterVolumeSpecName: "model-cache") pod "18398b06-3eed-4089-a429-61973020603b" (UID: "18398b06-3eed-4089-a429-61973020603b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:15:38.686599 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.686539    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-home" (OuterVolumeSpecName: "home") pod "18398b06-3eed-4089-a429-61973020603b" (UID: "18398b06-3eed-4089-a429-61973020603b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:15:38.686698 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.686591    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18398b06-3eed-4089-a429-61973020603b" (UID: "18398b06-3eed-4089-a429-61973020603b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:15:38.688573 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.688548    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-dshm" (OuterVolumeSpecName: "dshm") pod "18398b06-3eed-4089-a429-61973020603b" (UID: "18398b06-3eed-4089-a429-61973020603b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:15:38.688727 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.688627    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18398b06-3eed-4089-a429-61973020603b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18398b06-3eed-4089-a429-61973020603b" (UID: "18398b06-3eed-4089-a429-61973020603b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:15:38.688727 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.688670    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18398b06-3eed-4089-a429-61973020603b-kube-api-access-5qh5z" (OuterVolumeSpecName: "kube-api-access-5qh5z") pod "18398b06-3eed-4089-a429-61973020603b" (UID: "18398b06-3eed-4089-a429-61973020603b"). InnerVolumeSpecName "kube-api-access-5qh5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:15:38.787404 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.787360    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18398b06-3eed-4089-a429-61973020603b-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:15:38.787404 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.787400    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:15:38.787404 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.787413    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:15:38.787663 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.787426    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qh5z\" (UniqueName: \"kubernetes.io/projected/18398b06-3eed-4089-a429-61973020603b-kube-api-access-5qh5z\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:15:38.787663 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.787440    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:15:38.787663 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:38.787452    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18398b06-3eed-4089-a429-61973020603b-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:15:39.307327 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:39.307227    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" event={"ID":"f1637746-ba8c-494c-adca-67c118ddabae","Type":"ContainerStarted","Data":"0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0"}
Sep 17 16:15:39.308753 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:39.308725    2555 generic.go:356] "Generic (PLEG): container finished" podID="18398b06-3eed-4089-a429-61973020603b" containerID="daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22" exitCode=0
Sep 17 16:15:39.308915 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:39.308767    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"
Sep 17 16:15:39.308915 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:39.308803    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w" event={"ID":"18398b06-3eed-4089-a429-61973020603b","Type":"ContainerDied","Data":"daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22"}
Sep 17 16:15:39.308915 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:39.308887    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w" event={"ID":"18398b06-3eed-4089-a429-61973020603b","Type":"ContainerDied","Data":"ed29b009180ed94cd0fc6419b503a16aaaf4141fb4acfb05ca39c2c6ac3423a3"}
Sep 17 16:15:39.309042 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:39.308917    2555 scope.go:117] "RemoveContainer" containerID="daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22"
Sep 17 16:15:39.319114 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:39.319091    2555 scope.go:117] "RemoveContainer" containerID="daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22"
Sep 17 16:15:39.319502 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:15:39.319458    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22\": container with ID starting with daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22 not found: ID does not exist" containerID="daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22"
Sep 17 16:15:39.319585 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:39.319511    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22"} err="failed to get container status \"daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22\": rpc error: code = NotFound desc = could not find container \"daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22\": container with ID starting with daf6c0dd86aa9a2e10393fdf802962b774dd862f6b8ac8f2873b547b621bdd22 not found: ID does not exist"
Sep 17 16:15:39.334652 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:39.331722    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"]
Sep 17 16:15:39.337648 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:39.337612    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vg82w"]
Sep 17 16:15:40.319588 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:40.319500    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" event={"ID":"f1637746-ba8c-494c-adca-67c118ddabae","Type":"ContainerStarted","Data":"827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215"}
Sep 17 16:15:40.351722 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:40.351657    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podStartSLOduration=2.335240708 podStartE2EDuration="40.351634552s" totalImagesPullingTime="36.930278635s" totalInitContainerRuntime="1.086115209s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:15:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:15:40.347467205 +0000 UTC m=+2222.238327749" watchObservedRunningTime="2026-09-17 16:15:40.351634552 +0000 UTC m=+2222.242495095"
Sep 17 16:15:40.500623 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:40.500581    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:40.500812 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:40.500668    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:15:40.502274 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:40.502236    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:15:40.694747 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:40.694707    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18398b06-3eed-4089-a429-61973020603b" path="/var/lib/kubelet/pods/18398b06-3eed-4089-a429-61973020603b/volumes"
Sep 17 16:15:50.501436 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:15:50.501387    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:16:00.501359 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:16:00.501242    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:16:10.500897 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:16:10.500854    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:16:20.501312 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:16:20.501251    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:16:30.501282 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:16:30.501236    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:16:40.501751 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:16:40.501701    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:16:50.502156 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:16:50.502102    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:17:00.501048 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:00.501002    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:17:10.501039 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:10.500984    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:17:20.501761 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:20.501706    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:17:30.501109 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:30.500999    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 16:17:40.511668 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:40.511631    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:17:40.520216 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:40.520182    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:17:45.710735 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:45.710693    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"]
Sep 17 16:17:45.711338 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:45.711038    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="modelcar" containerID="cri-o://827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215" gracePeriod=60
Sep 17 16:17:45.993515 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:45.993401    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h_f1637746-ba8c-494c-adca-67c118ddabae/modelcar/0.log"
Sep 17 16:17:45.994551 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:45.994512    2555 generic.go:356] "Generic (PLEG): container finished" podID="f1637746-ba8c-494c-adca-67c118ddabae" containerID="827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215" exitCode=143
Sep 17 16:17:45.994727 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:17:45.994686    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" event={"ID":"f1637746-ba8c-494c-adca-67c118ddabae","Type":"ContainerDied","Data":"827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215"}
Sep 17 16:18:00.739068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:00.739012    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" containerID="cri-o://0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0" gracePeriod=45
Sep 17 16:18:01.597707 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.597681    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h_f1637746-ba8c-494c-adca-67c118ddabae/modelcar/0.log"
Sep 17 16:18:01.599662 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.599640    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:18:01.700613 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.700567    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42zjk\" (UniqueName: \"kubernetes.io/projected/f1637746-ba8c-494c-adca-67c118ddabae-kube-api-access-42zjk\") pod \"f1637746-ba8c-494c-adca-67c118ddabae\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") "
Sep 17 16:18:01.700613 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.700608    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-home\") pod \"f1637746-ba8c-494c-adca-67c118ddabae\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") "
Sep 17 16:18:01.700923 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.700643    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-tmp-dir\") pod \"f1637746-ba8c-494c-adca-67c118ddabae\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") "
Sep 17 16:18:01.700923 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.700660    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-dshm\") pod \"f1637746-ba8c-494c-adca-67c118ddabae\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") "
Sep 17 16:18:01.700923 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.700779    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-model-cache\") pod \"f1637746-ba8c-494c-adca-67c118ddabae\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") "
Sep 17 16:18:01.700923 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.700872    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1637746-ba8c-494c-adca-67c118ddabae-tls-certs\") pod \"f1637746-ba8c-494c-adca-67c118ddabae\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") "
Sep 17 16:18:01.700923 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.700909    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-kserve-provision-location\") pod \"f1637746-ba8c-494c-adca-67c118ddabae\" (UID: \"f1637746-ba8c-494c-adca-67c118ddabae\") "
Sep 17 16:18:01.701195 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.701041    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1637746-ba8c-494c-adca-67c118ddabae" (UID: "f1637746-ba8c-494c-adca-67c118ddabae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:18:01.701255 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.701193    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:18:01.701394 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.701340    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1637746-ba8c-494c-adca-67c118ddabae" (UID: "f1637746-ba8c-494c-adca-67c118ddabae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:18:01.701462 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.701422    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-home" (OuterVolumeSpecName: "home") pod "f1637746-ba8c-494c-adca-67c118ddabae" (UID: "f1637746-ba8c-494c-adca-67c118ddabae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:18:01.703216 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.703180    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-dshm" (OuterVolumeSpecName: "dshm") pod "f1637746-ba8c-494c-adca-67c118ddabae" (UID: "f1637746-ba8c-494c-adca-67c118ddabae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:18:01.703432 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.703410    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1637746-ba8c-494c-adca-67c118ddabae-kube-api-access-42zjk" (OuterVolumeSpecName: "kube-api-access-42zjk") pod "f1637746-ba8c-494c-adca-67c118ddabae" (UID: "f1637746-ba8c-494c-adca-67c118ddabae"). InnerVolumeSpecName "kube-api-access-42zjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:01.703482 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.703416    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1637746-ba8c-494c-adca-67c118ddabae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1637746-ba8c-494c-adca-67c118ddabae" (UID: "f1637746-ba8c-494c-adca-67c118ddabae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:18:01.711671 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.711640    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1637746-ba8c-494c-adca-67c118ddabae" (UID: "f1637746-ba8c-494c-adca-67c118ddabae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:18:01.801735 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.801691    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:18:01.801735 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.801723    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:18:01.801735 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.801732    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1637746-ba8c-494c-adca-67c118ddabae-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:18:01.801735 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.801741    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-kserve-provision-location\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:18:01.801735 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.801749    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42zjk\" (UniqueName: \"kubernetes.io/projected/f1637746-ba8c-494c-adca-67c118ddabae-kube-api-access-42zjk\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:18:01.802310 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:01.801759    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1637746-ba8c-494c-adca-67c118ddabae-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:18:02.073750 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.073662    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h_f1637746-ba8c-494c-adca-67c118ddabae/modelcar/0.log"
Sep 17 16:18:02.074427 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.074394    2555 generic.go:356] "Generic (PLEG): container finished" podID="f1637746-ba8c-494c-adca-67c118ddabae" containerID="0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0" exitCode=0
Sep 17 16:18:02.074572 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.074463    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"
Sep 17 16:18:02.074572 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.074488    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" event={"ID":"f1637746-ba8c-494c-adca-67c118ddabae","Type":"ContainerDied","Data":"0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0"}
Sep 17 16:18:02.074686 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.074569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" event={"ID":"f1637746-ba8c-494c-adca-67c118ddabae","Type":"ContainerDied","Data":"4928a2129086ad3db0c35ecc296f15bedb9c87ff7080e456845488a3eede44cf"}
Sep 17 16:18:02.074686 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.074610    2555 scope.go:117] "RemoveContainer" containerID="827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215"
Sep 17 16:18:02.088565 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.088538    2555 scope.go:117] "RemoveContainer" containerID="0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0"
Sep 17 16:18:02.097689 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.097658    2555 scope.go:117] "RemoveContainer" containerID="207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8"
Sep 17 16:18:02.106269 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.106228    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"]
Sep 17 16:18:02.106983 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.106963    2555 scope.go:117] "RemoveContainer" containerID="827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215"
Sep 17 16:18:02.107335 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:18:02.107315    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215\": container with ID starting with 827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215 not found: ID does not exist" containerID="827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215"
Sep 17 16:18:02.107396 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.107348    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215"} err="failed to get container status \"827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215\": rpc error: code = NotFound desc = could not find container \"827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215\": container with ID starting with 827eeac519355c322871cbb11648a4760436006f28c6185a61b284b962812215 not found: ID does not exist"
Sep 17 16:18:02.107396 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.107367    2555 scope.go:117] "RemoveContainer" containerID="0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0"
Sep 17 16:18:02.107670 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:18:02.107650    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0\": container with ID starting with 0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0 not found: ID does not exist" containerID="0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0"
Sep 17 16:18:02.107745 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.107677    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0"} err="failed to get container status \"0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0\": rpc error: code = NotFound desc = could not find container \"0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0\": container with ID starting with 0d5c620d04dec03b804eaf2d5cd0bdf079020852a7e0ad3d4c7c6abcf9b841d0 not found: ID does not exist"
Sep 17 16:18:02.107745 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.107700    2555 scope.go:117] "RemoveContainer" containerID="207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8"
Sep 17 16:18:02.108082 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:18:02.108058    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8\": container with ID starting with 207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8 not found: ID does not exist" containerID="207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8"
Sep 17 16:18:02.108145 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.108090    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8"} err="failed to get container status \"207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8\": rpc error: code = NotFound desc = could not find container \"207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8\": container with ID starting with 207738a163803f6bab70efe4b84d8ceb341f552ff2f71a158e938d75c9a145d8 not found: ID does not exist"
Sep 17 16:18:02.116977 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.116941    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h"]
Sep 17 16:18:02.513113 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.513065    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dck55h" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 16:18:02.695150 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:02.695113    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1637746-ba8c-494c-adca-67c118ddabae" path="/var/lib/kubelet/pods/f1637746-ba8c-494c-adca-67c118ddabae/volumes"
Sep 17 16:18:38.856046 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:38.856014    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:18:38.859807 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:38.859781    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:18:43.999958 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:43.999918    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"]
Sep 17 16:18:44.000517 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000384    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="modelcar-init"
Sep 17 16:18:44.000517 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000403    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="modelcar-init"
Sep 17 16:18:44.000517 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000416    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main"
Sep 17 16:18:44.000517 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000424    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main"
Sep 17 16:18:44.000517 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000448    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18398b06-3eed-4089-a429-61973020603b" containerName="main"
Sep 17 16:18:44.000517 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000456    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18398b06-3eed-4089-a429-61973020603b" containerName="main"
Sep 17 16:18:44.000517 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000463    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="modelcar"
Sep 17 16:18:44.000517 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000471    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="modelcar"
Sep 17 16:18:44.000956 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000572    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="main"
Sep 17 16:18:44.000956 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000583    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="18398b06-3eed-4089-a429-61973020603b" containerName="main"
Sep 17 16:18:44.000956 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.000593    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1637746-ba8c-494c-adca-67c118ddabae" containerName="modelcar"
Sep 17 16:18:44.003356 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.003328    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:18:44.007355 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.007328    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 16:18:44.007509 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.007330    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 16:18:44.007509 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.007341    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-9njkc\""
Sep 17 16:18:44.007509 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.007338    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-f5fzg\""
Sep 17 16:18:44.007509 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.007350    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 16:18:44.023942 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.023857    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"]
Sep 17 16:18:44.147784 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.147738    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3badb933-fdb6-41fe-9bb4-82e85f634c8b-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr\" (UID: \"3badb933-fdb6-41fe-9bb4-82e85f634c8b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:18:44.148011 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.147891    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czc6\" (UniqueName: \"kubernetes.io/projected/3badb933-fdb6-41fe-9bb4-82e85f634c8b-kube-api-access-6czc6\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr\" (UID: \"3badb933-fdb6-41fe-9bb4-82e85f634c8b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:18:44.249258 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.249217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3badb933-fdb6-41fe-9bb4-82e85f634c8b-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr\" (UID: \"3badb933-fdb6-41fe-9bb4-82e85f634c8b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:18:44.249492 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.249295    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czc6\" (UniqueName: \"kubernetes.io/projected/3badb933-fdb6-41fe-9bb4-82e85f634c8b-kube-api-access-6czc6\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr\" (UID: \"3badb933-fdb6-41fe-9bb4-82e85f634c8b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:18:44.252026 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.251956    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3badb933-fdb6-41fe-9bb4-82e85f634c8b-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr\" (UID: \"3badb933-fdb6-41fe-9bb4-82e85f634c8b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:18:44.261260 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.261231    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czc6\" (UniqueName: \"kubernetes.io/projected/3badb933-fdb6-41fe-9bb4-82e85f634c8b-kube-api-access-6czc6\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr\" (UID: \"3badb933-fdb6-41fe-9bb4-82e85f634c8b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:18:44.314585 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.314543    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:18:44.607269 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:44.607242    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"]
Sep 17 16:18:44.610331 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:18:44.610295    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3badb933_fdb6_41fe_9bb4_82e85f634c8b.slice/crio-3d98e096587442a2e54a46eacb42835a6c257aff5dcdf201cfb50f529793733a WatchSource:0}: Error finding container 3d98e096587442a2e54a46eacb42835a6c257aff5dcdf201cfb50f529793733a: Status 404 returned error can't find the container with id 3d98e096587442a2e54a46eacb42835a6c257aff5dcdf201cfb50f529793733a
Sep 17 16:18:45.292265 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:45.292231    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr" event={"ID":"3badb933-fdb6-41fe-9bb4-82e85f634c8b","Type":"ContainerStarted","Data":"9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663"}
Sep 17 16:18:45.292706 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:45.292284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr" event={"ID":"3badb933-fdb6-41fe-9bb4-82e85f634c8b","Type":"ContainerStarted","Data":"3d98e096587442a2e54a46eacb42835a6c257aff5dcdf201cfb50f529793733a"}
Sep 17 16:18:45.292706 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:45.292333    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:18:45.328620 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:18:45.328563    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr" podStartSLOduration=2.328543156 podStartE2EDuration="2.328543156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:45.32714008 +0000 UTC m=+2407.218000625" watchObservedRunningTime="2026-09-17 16:18:45.328543156 +0000 UTC m=+2407.219403700"
Sep 17 16:19:16.299279 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:19:16.299248    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:20:50.477523 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:20:50.477433    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"]
Sep 17 16:20:56.299226 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:20:56.299185    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr" podUID="3badb933-fdb6-41fe-9bb4-82e85f634c8b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:21:05.498810 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:05.498749    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr" podUID="3badb933-fdb6-41fe-9bb4-82e85f634c8b" containerName="main" containerID="cri-o://9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663" gracePeriod=45
Sep 17 16:21:06.299761 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:06.299720    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr" podUID="3badb933-fdb6-41fe-9bb4-82e85f634c8b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:21:16.299226 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:16.299182    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr" podUID="3badb933-fdb6-41fe-9bb4-82e85f634c8b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:21:16.299629 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:16.299303    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:21:26.299758 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:26.299718    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr" podUID="3badb933-fdb6-41fe-9bb4-82e85f634c8b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:21:35.746582 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:35.746556    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:21:35.875562 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:35.875524    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6czc6\" (UniqueName: \"kubernetes.io/projected/3badb933-fdb6-41fe-9bb4-82e85f634c8b-kube-api-access-6czc6\") pod \"3badb933-fdb6-41fe-9bb4-82e85f634c8b\" (UID: \"3badb933-fdb6-41fe-9bb4-82e85f634c8b\") "
Sep 17 16:21:35.875867 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:35.875654    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3badb933-fdb6-41fe-9bb4-82e85f634c8b-tls-certs\") pod \"3badb933-fdb6-41fe-9bb4-82e85f634c8b\" (UID: \"3badb933-fdb6-41fe-9bb4-82e85f634c8b\") "
Sep 17 16:21:35.878138 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:35.878098    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3badb933-fdb6-41fe-9bb4-82e85f634c8b-kube-api-access-6czc6" (OuterVolumeSpecName: "kube-api-access-6czc6") pod "3badb933-fdb6-41fe-9bb4-82e85f634c8b" (UID: "3badb933-fdb6-41fe-9bb4-82e85f634c8b"). InnerVolumeSpecName "kube-api-access-6czc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:21:35.878262 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:35.878160    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3badb933-fdb6-41fe-9bb4-82e85f634c8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3badb933-fdb6-41fe-9bb4-82e85f634c8b" (UID: "3badb933-fdb6-41fe-9bb4-82e85f634c8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:21:35.976414 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:35.976372    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3badb933-fdb6-41fe-9bb4-82e85f634c8b-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:21:35.976414 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:35.976407    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6czc6\" (UniqueName: \"kubernetes.io/projected/3badb933-fdb6-41fe-9bb4-82e85f634c8b-kube-api-access-6czc6\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:21:36.139580 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:36.139479    2555 generic.go:356] "Generic (PLEG): container finished" podID="3badb933-fdb6-41fe-9bb4-82e85f634c8b" containerID="9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663" exitCode=0
Sep 17 16:21:36.139580 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:36.139532    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"
Sep 17 16:21:36.139580 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:36.139563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr" event={"ID":"3badb933-fdb6-41fe-9bb4-82e85f634c8b","Type":"ContainerDied","Data":"9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663"}
Sep 17 16:21:36.139880 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:36.139621    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr" event={"ID":"3badb933-fdb6-41fe-9bb4-82e85f634c8b","Type":"ContainerDied","Data":"3d98e096587442a2e54a46eacb42835a6c257aff5dcdf201cfb50f529793733a"}
Sep 17 16:21:36.139880 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:36.139645    2555 scope.go:117] "RemoveContainer" containerID="9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663"
Sep 17 16:21:36.149050 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:36.149030    2555 scope.go:117] "RemoveContainer" containerID="9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663"
Sep 17 16:21:36.149373 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:36.149351    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663\": container with ID starting with 9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663 not found: ID does not exist" containerID="9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663"
Sep 17 16:21:36.149444 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:36.149389    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663"} err="failed to get container status \"9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663\": rpc error: code = NotFound desc = could not find container \"9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663\": container with ID starting with 9003808ec3b3e4ac5307866e156d29124f519b09675cdef185e3c769de8b6663 not found: ID does not exist"
Sep 17 16:21:36.161632 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:36.161595    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"]
Sep 17 16:21:36.167248 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:36.167218    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58667pr"]
Sep 17 16:21:36.694697 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:36.694666    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3badb933-fdb6-41fe-9bb4-82e85f634c8b" path="/var/lib/kubelet/pods/3badb933-fdb6-41fe-9bb4-82e85f634c8b/volumes"
Sep 17 16:21:40.600074 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.600041    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"]
Sep 17 16:21:40.600534 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.600502    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3badb933-fdb6-41fe-9bb4-82e85f634c8b" containerName="main"
Sep 17 16:21:40.600534 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.600521    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3badb933-fdb6-41fe-9bb4-82e85f634c8b" containerName="main"
Sep 17 16:21:40.600665 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.600627    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3badb933-fdb6-41fe-9bb4-82e85f634c8b" containerName="main"
Sep 17 16:21:40.603425 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.603402    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.608970 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:40.608896    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-028f7809\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.608970 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:40.608918    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-gw-ca-root-cert\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-028f7809\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap"
Sep 17 16:21:40.608970 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:40.608918    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-gateway-1-openshift-default-dockercfg-59c6b\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-028f7809\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-59c6b\"" type="*v1.Secret"
Sep 17 16:21:40.609240 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:40.609009    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-028f7809\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 17 16:21:40.609288 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:40.609238    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-243.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-028f7809\": no relationship found between node 'ip-10-0-135-243.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 17 16:21:40.621942 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.621912    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"]
Sep 17 16:21:40.713815 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.713768    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.714068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.713845    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.714068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.713885    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.714068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.713917    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.714068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.713950    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.714068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.713978    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.714068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.714046    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.714400 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.714090    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rpb8\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-kube-api-access-9rpb8\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.714400 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.714117    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815074 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815042    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815281 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815281 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815122    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rpb8\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-kube-api-access-9rpb8\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815281 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815143    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815281 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815215    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815281 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815233    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815281 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815266    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815623 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815294    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815623 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815323    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815777 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815749    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815777 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815955 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815935    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.815996 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.815971    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.818213 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.818190    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.818369 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.818351    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:40.825331 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:40.825305    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:41.546995 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:41.546954    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-59c6b\""
Sep 17 16:21:41.811332 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:41.811229    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 16:21:41.815713 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:41.815680    2555 configmap.go:193] Couldn't get configMap e2e-test-llm-inference-service-028f7809/openshift-gw-ca-root-cert: failed to sync configmap cache: timed out waiting for the condition
Sep 17 16:21:41.815883 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:41.815771    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istiod-ca-cert podName:3b70d873-edbd-44a2-89d3-aa9369fbefa5 nodeName:}" failed. No retries permitted until 2026-09-17 16:21:42.315754528 +0000 UTC m=+2584.206615055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "istiod-ca-cert" (UniqueName: "kubernetes.io/configmap/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istiod-ca-cert") pod "router-gateway-1-openshift-default-cc7cb45dd-x4p6r" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 16:21:41.818151 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:41.818132    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 17 16:21:41.822795 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:41.822779    2555 projected.go:291] Couldn't get configMap e2e-test-llm-inference-service-028f7809/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 17 16:21:41.822861 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:41.822813    2555 projected.go:196] Error preparing data for projected volume kube-api-access-9rpb8 for pod e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r: failed to sync configmap cache: timed out waiting for the condition
Sep 17 16:21:41.822900 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:21:41.822885    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-kube-api-access-9rpb8 podName:3b70d873-edbd-44a2-89d3-aa9369fbefa5 nodeName:}" failed. No retries permitted until 2026-09-17 16:21:42.322870519 +0000 UTC m=+2584.213731041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9rpb8" (UniqueName: "kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-kube-api-access-9rpb8") pod "router-gateway-1-openshift-default-cc7cb45dd-x4p6r" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 16:21:42.073948 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:42.073857    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 16:21:42.329920 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:42.329874    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rpb8\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-kube-api-access-9rpb8\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:42.330156 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:42.329971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:42.330616 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:42.330595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:42.332508 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:42.332478    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rpb8\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-kube-api-access-9rpb8\") pod \"router-gateway-1-openshift-default-cc7cb45dd-x4p6r\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:42.414115 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:42.414070    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:42.700775 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:21:42.700734    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b70d873_edbd_44a2_89d3_aa9369fbefa5.slice/crio-9a92031b32d51dfa899de846e1c93ecdfae10391fbc722f387271f66b6ea84ca WatchSource:0}: Error finding container 9a92031b32d51dfa899de846e1c93ecdfae10391fbc722f387271f66b6ea84ca: Status 404 returned error can't find the container with id 9a92031b32d51dfa899de846e1c93ecdfae10391fbc722f387271f66b6ea84ca
Sep 17 16:21:42.701492 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:42.701464    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"]
Sep 17 16:21:42.703244 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:42.703214    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:21:42.703335 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:42.703279    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:21:42.703335 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:42.703310    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:21:43.175729 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:43.175690    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" event={"ID":"3b70d873-edbd-44a2-89d3-aa9369fbefa5","Type":"ContainerStarted","Data":"73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5"}
Sep 17 16:21:43.176288 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:43.175751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" event={"ID":"3b70d873-edbd-44a2-89d3-aa9369fbefa5","Type":"ContainerStarted","Data":"9a92031b32d51dfa899de846e1c93ecdfae10391fbc722f387271f66b6ea84ca"}
Sep 17 16:21:43.195937 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:43.195865    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" podStartSLOduration=3.195841247 podStartE2EDuration="3.195841247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:21:43.194316581 +0000 UTC m=+2585.085177126" watchObservedRunningTime="2026-09-17 16:21:43.195841247 +0000 UTC m=+2585.086701786"
Sep 17 16:21:43.414752 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:43.414707    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:43.416382 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:43.416344    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 16:21:44.414817 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:44.414708    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 16:21:45.415313 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:45.415264    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 16:21:46.415243 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:46.415197    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 16:21:47.415023 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:47.414979    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 16:21:48.415547 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:48.415494    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 16:21:49.418535 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:49.418495    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:49.419069 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:49.418785    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:21:49.419463 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:21:49.419444    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:22:33.060074 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.060032    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"]
Sep 17 16:22:33.062893 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.062871    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:22:33.066805 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.066778    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 16:22:33.067191 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.067166    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-dmggr\""
Sep 17 16:22:33.067346 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.067209    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 16:22:33.067346 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.067169    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 16:22:33.067346 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.067254    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-ns7z5\""
Sep 17 16:22:33.073890 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.073864    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"]
Sep 17 16:22:33.157778 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.157732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e23958-b660-4584-a700-b2b490d6c07f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46\" (UID: \"02e23958-b660-4584-a700-b2b490d6c07f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:22:33.158000 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.157812    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlk75\" (UniqueName: \"kubernetes.io/projected/02e23958-b660-4584-a700-b2b490d6c07f-kube-api-access-hlk75\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46\" (UID: \"02e23958-b660-4584-a700-b2b490d6c07f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:22:33.259112 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.259059    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlk75\" (UniqueName: \"kubernetes.io/projected/02e23958-b660-4584-a700-b2b490d6c07f-kube-api-access-hlk75\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46\" (UID: \"02e23958-b660-4584-a700-b2b490d6c07f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:22:33.259357 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.259231    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e23958-b660-4584-a700-b2b490d6c07f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46\" (UID: \"02e23958-b660-4584-a700-b2b490d6c07f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:22:33.261941 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.261916    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e23958-b660-4584-a700-b2b490d6c07f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46\" (UID: \"02e23958-b660-4584-a700-b2b490d6c07f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:22:33.268509 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.268485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlk75\" (UniqueName: \"kubernetes.io/projected/02e23958-b660-4584-a700-b2b490d6c07f-kube-api-access-hlk75\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46\" (UID: \"02e23958-b660-4584-a700-b2b490d6c07f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:22:33.375627 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.375587    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:22:33.649985 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:33.649956    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"]
Sep 17 16:22:33.652192 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:22:33.652147    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e23958_b660_4584_a700_b2b490d6c07f.slice/crio-6a770051ef57bb91d207c7c0639409bb35f7b37fafc595f5055112a09907b801 WatchSource:0}: Error finding container 6a770051ef57bb91d207c7c0639409bb35f7b37fafc595f5055112a09907b801: Status 404 returned error can't find the container with id 6a770051ef57bb91d207c7c0639409bb35f7b37fafc595f5055112a09907b801
Sep 17 16:22:34.446037 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:34.445993    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" event={"ID":"02e23958-b660-4584-a700-b2b490d6c07f","Type":"ContainerStarted","Data":"c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0"}
Sep 17 16:22:34.446662 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:34.446633    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" event={"ID":"02e23958-b660-4584-a700-b2b490d6c07f","Type":"ContainerStarted","Data":"6a770051ef57bb91d207c7c0639409bb35f7b37fafc595f5055112a09907b801"}
Sep 17 16:22:34.447598 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:34.447575    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:22:34.467790 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:22:34.467717    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" podStartSLOduration=1.467690963 podStartE2EDuration="1.467690963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:22:34.465286953 +0000 UTC m=+2636.356147528" watchObservedRunningTime="2026-09-17 16:22:34.467690963 +0000 UTC m=+2636.358551508"
Sep 17 16:23:06.462421 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:23:06.462389    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:23:38.888023 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:23:38.887930    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:23:38.893149 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:23:38.893125    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:24:35.178661 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:24:35.178605    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"]
Sep 17 16:24:36.463205 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:24:36.463168    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" podUID="02e23958-b660-4584-a700-b2b490d6c07f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:24:46.462916 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:24:46.462873    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" podUID="02e23958-b660-4584-a700-b2b490d6c07f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:24:50.198880 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:24:50.198819    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" podUID="02e23958-b660-4584-a700-b2b490d6c07f" containerName="main" containerID="cri-o://c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0" gracePeriod=45
Sep 17 16:24:55.815127 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:24:55.815080    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"]
Sep 17 16:24:55.815565 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:24:55.815282    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerName="istio-proxy" containerID="cri-o://73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5" gracePeriod=30
Sep 17 16:24:56.462544 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:24:56.462505    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" podUID="02e23958-b660-4584-a700-b2b490d6c07f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:24:56.462727 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:24:56.462616    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:25:01.069751 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.069718    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:25:01.115794 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.115714    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-podinfo\") pod \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") "
Sep 17 16:25:01.115794 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.115757    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-credential-socket\") pod \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") "
Sep 17 16:25:01.116022 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.115878    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-token\") pod \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") "
Sep 17 16:25:01.116022 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.115915    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-certs\") pod \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") "
Sep 17 16:25:01.116022 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.115940    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-data\") pod \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") "
Sep 17 16:25:01.116022 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.115969    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rpb8\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-kube-api-access-9rpb8\") pod \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") "
Sep 17 16:25:01.116022 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.116000    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istiod-ca-cert\") pod \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") "
Sep 17 16:25:01.116284 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.116126    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b70d873-edbd-44a2-89d3-aa9369fbefa5" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:25:01.116284 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.116184    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-envoy\") pod \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") "
Sep 17 16:25:01.116375 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.116267    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b70d873-edbd-44a2-89d3-aa9369fbefa5" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:25:01.116432 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.116403    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b70d873-edbd-44a2-89d3-aa9369fbefa5" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:25:01.116518 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.116499    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-socket\") pod \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\" (UID: \"3b70d873-edbd-44a2-89d3-aa9369fbefa5\") "
Sep 17 16:25:01.116771 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.116748    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b70d873-edbd-44a2-89d3-aa9369fbefa5" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:25:01.116907 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.116646    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b70d873-edbd-44a2-89d3-aa9369fbefa5" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:25:01.117141 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.117120    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:01.117213 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.117146    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-data\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:01.117213 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.117160    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istiod-ca-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:01.117213 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.117173    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-workload-socket\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:01.117213 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.117185    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-credential-socket\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:01.118358 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.118330    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b70d873-edbd-44a2-89d3-aa9369fbefa5" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:25:01.118476 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.118375    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-kube-api-access-9rpb8" (OuterVolumeSpecName: "kube-api-access-9rpb8") pod "3b70d873-edbd-44a2-89d3-aa9369fbefa5" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5"). InnerVolumeSpecName "kube-api-access-9rpb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:25:01.118540 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.118492    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b70d873-edbd-44a2-89d3-aa9369fbefa5" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 16:25:01.118540 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.118498    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b70d873-edbd-44a2-89d3-aa9369fbefa5" (UID: "3b70d873-edbd-44a2-89d3-aa9369fbefa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:25:01.189490 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.189453    2555 generic.go:356] "Generic (PLEG): container finished" podID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerID="73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5" exitCode=0
Sep 17 16:25:01.189647 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.189491    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"
Sep 17 16:25:01.189647 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.189505    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" event={"ID":"3b70d873-edbd-44a2-89d3-aa9369fbefa5","Type":"ContainerDied","Data":"73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5"}
Sep 17 16:25:01.189647 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.189579    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r" event={"ID":"3b70d873-edbd-44a2-89d3-aa9369fbefa5","Type":"ContainerDied","Data":"9a92031b32d51dfa899de846e1c93ecdfae10391fbc722f387271f66b6ea84ca"}
Sep 17 16:25:01.189647 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.189615    2555 scope.go:117] "RemoveContainer" containerID="73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5"
Sep 17 16:25:01.200071 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.200048    2555 scope.go:117] "RemoveContainer" containerID="73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5"
Sep 17 16:25:01.200414 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:25:01.200391    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5\": container with ID starting with 73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5 not found: ID does not exist" containerID="73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5"
Sep 17 16:25:01.200468 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.200423    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5"} err="failed to get container status \"73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5\": rpc error: code = NotFound desc = could not find container \"73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5\": container with ID starting with 73902ee4a487304f4a812bfffd6f6df6aa020f7e7fe4a0ef32ceb4abf4a080c5 not found: ID does not exist"
Sep 17 16:25:01.218264 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.218232    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-envoy\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:01.218264 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.218259    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-podinfo\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:01.218264 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.218268    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-istio-token\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:01.218483 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.218277    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rpb8\" (UniqueName: \"kubernetes.io/projected/3b70d873-edbd-44a2-89d3-aa9369fbefa5-kube-api-access-9rpb8\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:01.234366 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.234331    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"]
Sep 17 16:25:01.244595 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:01.244567    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-x4p6r"]
Sep 17 16:25:02.694283 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:02.694248    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" path="/var/lib/kubelet/pods/3b70d873-edbd-44a2-89d3-aa9369fbefa5/volumes"
Sep 17 16:25:06.462909 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:06.462869    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" podUID="02e23958-b660-4584-a700-b2b490d6c07f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:25:16.462486 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:16.462391    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" podUID="02e23958-b660-4584-a700-b2b490d6c07f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:25:20.445495 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:20.445464    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:25:20.581424 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:20.581382    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlk75\" (UniqueName: \"kubernetes.io/projected/02e23958-b660-4584-a700-b2b490d6c07f-kube-api-access-hlk75\") pod \"02e23958-b660-4584-a700-b2b490d6c07f\" (UID: \"02e23958-b660-4584-a700-b2b490d6c07f\") "
Sep 17 16:25:20.581631 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:20.581473    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e23958-b660-4584-a700-b2b490d6c07f-tls-certs\") pod \"02e23958-b660-4584-a700-b2b490d6c07f\" (UID: \"02e23958-b660-4584-a700-b2b490d6c07f\") "
Sep 17 16:25:20.583709 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:20.583680    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e23958-b660-4584-a700-b2b490d6c07f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02e23958-b660-4584-a700-b2b490d6c07f" (UID: "02e23958-b660-4584-a700-b2b490d6c07f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:25:20.583860 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:20.583713    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e23958-b660-4584-a700-b2b490d6c07f-kube-api-access-hlk75" (OuterVolumeSpecName: "kube-api-access-hlk75") pod "02e23958-b660-4584-a700-b2b490d6c07f" (UID: "02e23958-b660-4584-a700-b2b490d6c07f"). InnerVolumeSpecName "kube-api-access-hlk75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:25:20.682913 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:20.682871    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlk75\" (UniqueName: \"kubernetes.io/projected/02e23958-b660-4584-a700-b2b490d6c07f-kube-api-access-hlk75\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:20.682913 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:20.682908    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e23958-b660-4584-a700-b2b490d6c07f-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:25:21.288154 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:21.288120    2555 generic.go:356] "Generic (PLEG): container finished" podID="02e23958-b660-4584-a700-b2b490d6c07f" containerID="c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0" exitCode=0
Sep 17 16:25:21.288487 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:21.288170    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"
Sep 17 16:25:21.288487 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:21.288206    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" event={"ID":"02e23958-b660-4584-a700-b2b490d6c07f","Type":"ContainerDied","Data":"c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0"}
Sep 17 16:25:21.288487 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:21.288271    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46" event={"ID":"02e23958-b660-4584-a700-b2b490d6c07f","Type":"ContainerDied","Data":"6a770051ef57bb91d207c7c0639409bb35f7b37fafc595f5055112a09907b801"}
Sep 17 16:25:21.288487 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:21.288302    2555 scope.go:117] "RemoveContainer" containerID="c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0"
Sep 17 16:25:21.297010 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:21.296991    2555 scope.go:117] "RemoveContainer" containerID="c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0"
Sep 17 16:25:21.297340 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:25:21.297318    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0\": container with ID starting with c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0 not found: ID does not exist" containerID="c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0"
Sep 17 16:25:21.297386 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:21.297348    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0"} err="failed to get container status \"c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0\": rpc error: code = NotFound desc = could not find container \"c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0\": container with ID starting with c1f9a77ec08cb184f59ca9ba604038575f1193b15592f25a2bb6565a69d428a0 not found: ID does not exist"
Sep 17 16:25:21.320444 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:21.320404    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"]
Sep 17 16:25:21.327516 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:21.327487    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-l7b46"]
Sep 17 16:25:22.695213 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:22.695176    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e23958-b660-4584-a700-b2b490d6c07f" path="/var/lib/kubelet/pods/02e23958-b660-4584-a700-b2b490d6c07f/volumes"
Sep 17 16:25:24.925337 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.925307    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"]
Sep 17 16:25:24.925747 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.925604    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e23958-b660-4584-a700-b2b490d6c07f" containerName="main"
Sep 17 16:25:24.925747 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.925616    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e23958-b660-4584-a700-b2b490d6c07f" containerName="main"
Sep 17 16:25:24.925747 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.925630    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerName="istio-proxy"
Sep 17 16:25:24.925747 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.925636    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerName="istio-proxy"
Sep 17 16:25:24.925747 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.925693    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b70d873-edbd-44a2-89d3-aa9369fbefa5" containerName="istio-proxy"
Sep 17 16:25:24.925747 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.925704    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e23958-b660-4584-a700-b2b490d6c07f" containerName="main"
Sep 17 16:25:24.929963 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.929942    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:25:24.936769 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.936359    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-5xqtg\""
Sep 17 16:25:24.936769 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.936421    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 16:25:24.937062 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.937036    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-2sg2q\""
Sep 17 16:25:24.937156 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.937090    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 16:25:24.937313 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.937300    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 16:25:24.956074 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:24.956034    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"]
Sep 17 16:25:25.019567 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:25.019524    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx\" (UID: \"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:25:25.019785 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:25.019575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5gq\" (UniqueName: \"kubernetes.io/projected/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-kube-api-access-gj5gq\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx\" (UID: \"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:25:25.121176 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:25.121131    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx\" (UID: \"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:25:25.121176 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:25.121182    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5gq\" (UniqueName: \"kubernetes.io/projected/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-kube-api-access-gj5gq\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx\" (UID: \"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:25:25.123806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:25.123774    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx\" (UID: \"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:25:25.131946 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:25.131918    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5gq\" (UniqueName: \"kubernetes.io/projected/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-kube-api-access-gj5gq\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx\" (UID: \"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:25:25.240052 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:25.239956    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:25:25.531286 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:25.531250    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"]
Sep 17 16:25:25.532429 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:25:25.532397    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5496c28_b87c_4c10_b0cd_fe47dc1b2c51.slice/crio-23aeae5f303ec926848cf99f4bf18866941d454d58b689f262b88d3ff22586f6 WatchSource:0}: Error finding container 23aeae5f303ec926848cf99f4bf18866941d454d58b689f262b88d3ff22586f6: Status 404 returned error can't find the container with id 23aeae5f303ec926848cf99f4bf18866941d454d58b689f262b88d3ff22586f6
Sep 17 16:25:26.322951 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:26.322911    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" event={"ID":"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51","Type":"ContainerStarted","Data":"d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0"}
Sep 17 16:25:26.323377 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:26.322969    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" event={"ID":"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51","Type":"ContainerStarted","Data":"23aeae5f303ec926848cf99f4bf18866941d454d58b689f262b88d3ff22586f6"}
Sep 17 16:25:26.323377 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:26.323033    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:25:26.343786 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:26.343728    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" podStartSLOduration=2.343708704 podStartE2EDuration="2.343708704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:25:26.34252081 +0000 UTC m=+2808.233381356" watchObservedRunningTime="2026-09-17 16:25:26.343708704 +0000 UTC m=+2808.234569247"
Sep 17 16:25:57.330318 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:25:57.330281    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:27:54.213026 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:27:54.212983    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"]
Sep 17 16:27:57.330309 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:27:57.330272    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" podUID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:28:07.330574 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:07.330534    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" podUID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:28:09.232276 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:09.232231    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" podUID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" containerName="main" containerID="cri-o://d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0" gracePeriod=45
Sep 17 16:28:17.329642 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:17.329584    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" podUID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:28:17.330103 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:17.329706    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:28:27.330020 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:27.329977    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" podUID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:28:37.330519 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:37.330478    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" podUID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:28:38.923182 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:38.923152    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:28:38.927770 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:38.927744    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:28:39.486565 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:39.486539    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:28:39.615754 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:39.615702    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj5gq\" (UniqueName: \"kubernetes.io/projected/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-kube-api-access-gj5gq\") pod \"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51\" (UID: \"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51\") "
Sep 17 16:28:39.615995 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:39.615870    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-tls-certs\") pod \"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51\" (UID: \"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51\") "
Sep 17 16:28:39.618074 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:39.618043    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" (UID: "f5496c28-b87c-4c10-b0cd-fe47dc1b2c51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:39.618190 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:39.618079    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-kube-api-access-gj5gq" (OuterVolumeSpecName: "kube-api-access-gj5gq") pod "f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" (UID: "f5496c28-b87c-4c10-b0cd-fe47dc1b2c51"). InnerVolumeSpecName "kube-api-access-gj5gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:39.716888 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:39.716818    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:28:39.716888 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:39.716881    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj5gq\" (UniqueName: \"kubernetes.io/projected/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51-kube-api-access-gj5gq\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:28:40.303183 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:40.303151    2555 generic.go:356] "Generic (PLEG): container finished" podID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" containerID="d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0" exitCode=0
Sep 17 16:28:40.303632 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:40.303200    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"
Sep 17 16:28:40.303632 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:40.303241    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" event={"ID":"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51","Type":"ContainerDied","Data":"d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0"}
Sep 17 16:28:40.303632 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:40.303292    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx" event={"ID":"f5496c28-b87c-4c10-b0cd-fe47dc1b2c51","Type":"ContainerDied","Data":"23aeae5f303ec926848cf99f4bf18866941d454d58b689f262b88d3ff22586f6"}
Sep 17 16:28:40.303632 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:40.303318    2555 scope.go:117] "RemoveContainer" containerID="d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0"
Sep 17 16:28:40.312631 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:40.312606    2555 scope.go:117] "RemoveContainer" containerID="d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0"
Sep 17 16:28:40.312972 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:28:40.312951    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0\": container with ID starting with d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0 not found: ID does not exist" containerID="d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0"
Sep 17 16:28:40.313037 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:40.312986    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0"} err="failed to get container status \"d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0\": rpc error: code = NotFound desc = could not find container \"d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0\": container with ID starting with d893eed5e3060b594104c3c1e4d13d5f33ea7e6c5bdc234f7da37274c98e9fd0 not found: ID does not exist"
Sep 17 16:28:40.326221 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:40.326188    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"]
Sep 17 16:28:40.329573 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:40.329540    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemrswx"]
Sep 17 16:28:40.695008 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:40.694967    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" path="/var/lib/kubelet/pods/f5496c28-b87c-4c10-b0cd-fe47dc1b2c51/volumes"
Sep 17 16:28:47.758897 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.758865    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"]
Sep 17 16:28:47.759309 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.759156    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" containerName="main"
Sep 17 16:28:47.759309 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.759167    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" containerName="main"
Sep 17 16:28:47.759309 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.759221    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5496c28-b87c-4c10-b0cd-fe47dc1b2c51" containerName="main"
Sep 17 16:28:47.763509 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.763489    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.766161 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.766121    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:28:47.766420 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.766397    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 16:28:47.767002 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.766977    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 16:28:47.767107 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.767028    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 16:28:47.767107 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.767039    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 16:28:47.767107 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.767028    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-w8bl6\""
Sep 17 16:28:47.767524 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.767503    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-pwwxw\""
Sep 17 16:28:47.779229 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.779202    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"]
Sep 17 16:28:47.886957 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.886918    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-home\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.886957 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.886963    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16512504-2307-4f7d-9b85-57e298b7604a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.887197 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.886996    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmqg\" (UniqueName: \"kubernetes.io/projected/16512504-2307-4f7d-9b85-57e298b7604a-kube-api-access-5hmqg\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.887197 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.887054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.887197 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.887084    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16512504-2307-4f7d-9b85-57e298b7604a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.887197 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.887110    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-dshm\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.887197 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.887171    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.887197 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.887187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.987606 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.987562    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-home\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.987606 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.987608    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16512504-2307-4f7d-9b85-57e298b7604a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.987874 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.987732    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmqg\" (UniqueName: \"kubernetes.io/projected/16512504-2307-4f7d-9b85-57e298b7604a-kube-api-access-5hmqg\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.987874 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.987783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.987874 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.987832    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16512504-2307-4f7d-9b85-57e298b7604a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.987874 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.987859    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-dshm\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.988090 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.987902    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.988090 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.987919    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.988090 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.988018    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-home\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.988291 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.988259    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.988385 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.988292    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.988498 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.988474    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.988774 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.988719    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16512504-2307-4f7d-9b85-57e298b7604a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.990391 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.990363    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-dshm\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:47.990495 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:47.990458    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16512504-2307-4f7d-9b85-57e298b7604a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:48.004532 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:48.004504    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmqg\" (UniqueName: \"kubernetes.io/projected/16512504-2307-4f7d-9b85-57e298b7604a-kube-api-access-5hmqg\") pod \"custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:48.072898 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:48.072754    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:48.357318 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:48.357288    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"]
Sep 17 16:28:48.358959 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:28:48.358924    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16512504_2307_4f7d_9b85_57e298b7604a.slice/crio-aec1da052d1e8ca9b01c8845c8dfbdaba6d6faa67633c1e2a9503e9d3a82f4d8 WatchSource:0}: Error finding container aec1da052d1e8ca9b01c8845c8dfbdaba6d6faa67633c1e2a9503e9d3a82f4d8: Status 404 returned error can't find the container with id aec1da052d1e8ca9b01c8845c8dfbdaba6d6faa67633c1e2a9503e9d3a82f4d8
Sep 17 16:28:48.361262 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:48.361238    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:28:49.355314 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:49.355273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" event={"ID":"16512504-2307-4f7d-9b85-57e298b7604a","Type":"ContainerStarted","Data":"aec1da052d1e8ca9b01c8845c8dfbdaba6d6faa67633c1e2a9503e9d3a82f4d8"}
Sep 17 16:28:50.361568 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:50.361526    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" event={"ID":"16512504-2307-4f7d-9b85-57e298b7604a","Type":"ContainerStarted","Data":"e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e"}
Sep 17 16:28:50.362064 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:50.361653    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:28:54.386686 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:54.386646    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" event={"ID":"16512504-2307-4f7d-9b85-57e298b7604a","Type":"ContainerStarted","Data":"8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e"}
Sep 17 16:28:57.402972 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:57.402935    2555 generic.go:356] "Generic (PLEG): container finished" podID="16512504-2307-4f7d-9b85-57e298b7604a" containerID="8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e" exitCode=0
Sep 17 16:28:57.403423 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:57.403009    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" event={"ID":"16512504-2307-4f7d-9b85-57e298b7604a","Type":"ContainerDied","Data":"8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e"}
Sep 17 16:28:58.410994 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:58.410952    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" event={"ID":"16512504-2307-4f7d-9b85-57e298b7604a","Type":"ContainerStarted","Data":"02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26"}
Sep 17 16:28:58.438690 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:28:58.438624    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podStartSLOduration=3.497755546 podStartE2EDuration="11.438603709s" totalImagesPullingTime="5.59324928s" totalInitContainerRuntime="2.347598883s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:28:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:28:58.436996353 +0000 UTC m=+3020.327856909" watchObservedRunningTime="2026-09-17 16:28:58.438603709 +0000 UTC m=+3020.329464252"
Sep 17 16:29:08.073798 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:29:08.073745    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:29:08.074310 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:29:08.073867    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:29:08.075326 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:29:08.075289    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:29:08.089134 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:29:08.089107    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:29:18.073217 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:29:18.073163    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:29:28.074303 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:29:28.074255    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:29:38.073508 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:29:38.073406    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:29:48.073356 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:29:48.073312    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:29:58.073978 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:29:58.073930    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:30:08.074148 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:30:08.074095    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:30:18.074122 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:30:18.074060    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:30:28.073442 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:30:28.073389    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:30:38.073712 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:30:38.073655    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:30:48.073246 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:30:48.073197    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 17 16:30:58.083239 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:30:58.083200    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:30:58.096089 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:30:58.096050    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:31:07.972014 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:07.971971    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"]
Sep 17 16:31:22.997143 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:22.997094    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main" containerID="cri-o://02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26" gracePeriod=45
Sep 17 16:31:23.613632 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.613582    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="llm-d-routing-sidecar" containerID="cri-o://e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e" gracePeriod=45
Sep 17 16:31:23.863417 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.863384    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:31:23.922603 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.922568    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-dshm\") pod \"16512504-2307-4f7d-9b85-57e298b7604a\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") "
Sep 17 16:31:23.922816 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.922621    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-tmp-dir\") pod \"16512504-2307-4f7d-9b85-57e298b7604a\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") "
Sep 17 16:31:23.922816 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.922661    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-home\") pod \"16512504-2307-4f7d-9b85-57e298b7604a\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") "
Sep 17 16:31:23.922816 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.922678    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16512504-2307-4f7d-9b85-57e298b7604a-tls-certs\") pod \"16512504-2307-4f7d-9b85-57e298b7604a\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") "
Sep 17 16:31:23.922816 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.922718    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16512504-2307-4f7d-9b85-57e298b7604a-cabundle-cert\") pod \"16512504-2307-4f7d-9b85-57e298b7604a\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") "
Sep 17 16:31:23.922816 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.922754    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-model-cache\") pod \"16512504-2307-4f7d-9b85-57e298b7604a\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") "
Sep 17 16:31:23.922816 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.922775    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-kserve-provision-location\") pod \"16512504-2307-4f7d-9b85-57e298b7604a\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") "
Sep 17 16:31:23.922816 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.922803    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hmqg\" (UniqueName: \"kubernetes.io/projected/16512504-2307-4f7d-9b85-57e298b7604a-kube-api-access-5hmqg\") pod \"16512504-2307-4f7d-9b85-57e298b7604a\" (UID: \"16512504-2307-4f7d-9b85-57e298b7604a\") "
Sep 17 16:31:23.923225 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.923076    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16512504-2307-4f7d-9b85-57e298b7604a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "16512504-2307-4f7d-9b85-57e298b7604a" (UID: "16512504-2307-4f7d-9b85-57e298b7604a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:31:23.923225 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.923207    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-model-cache" (OuterVolumeSpecName: "model-cache") pod "16512504-2307-4f7d-9b85-57e298b7604a" (UID: "16512504-2307-4f7d-9b85-57e298b7604a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:23.923516 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.923482    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-home" (OuterVolumeSpecName: "home") pod "16512504-2307-4f7d-9b85-57e298b7604a" (UID: "16512504-2307-4f7d-9b85-57e298b7604a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:23.925220 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.925186    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-dshm" (OuterVolumeSpecName: "dshm") pod "16512504-2307-4f7d-9b85-57e298b7604a" (UID: "16512504-2307-4f7d-9b85-57e298b7604a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:23.925353 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.925215    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16512504-2307-4f7d-9b85-57e298b7604a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16512504-2307-4f7d-9b85-57e298b7604a" (UID: "16512504-2307-4f7d-9b85-57e298b7604a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:31:23.925416 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.925351    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16512504-2307-4f7d-9b85-57e298b7604a-kube-api-access-5hmqg" (OuterVolumeSpecName: "kube-api-access-5hmqg") pod "16512504-2307-4f7d-9b85-57e298b7604a" (UID: "16512504-2307-4f7d-9b85-57e298b7604a"). InnerVolumeSpecName "kube-api-access-5hmqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:31:23.936092 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.936056    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16512504-2307-4f7d-9b85-57e298b7604a" (UID: "16512504-2307-4f7d-9b85-57e298b7604a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:23.943663 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:23.943623    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16512504-2307-4f7d-9b85-57e298b7604a" (UID: "16512504-2307-4f7d-9b85-57e298b7604a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:24.024118 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.024081    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:31:24.024118 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.024116    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-kserve-provision-location\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:31:24.024565 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.024129    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hmqg\" (UniqueName: \"kubernetes.io/projected/16512504-2307-4f7d-9b85-57e298b7604a-kube-api-access-5hmqg\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:31:24.024565 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.024138    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:31:24.024565 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.024147    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:31:24.024565 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.024156    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16512504-2307-4f7d-9b85-57e298b7604a-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:31:24.024565 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.024165    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16512504-2307-4f7d-9b85-57e298b7604a-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:31:24.024565 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.024173    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16512504-2307-4f7d-9b85-57e298b7604a-cabundle-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:31:24.194644 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.194545    2555 generic.go:356] "Generic (PLEG): container finished" podID="16512504-2307-4f7d-9b85-57e298b7604a" containerID="02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26" exitCode=0
Sep 17 16:31:24.194644 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.194573    2555 generic.go:356] "Generic (PLEG): container finished" podID="16512504-2307-4f7d-9b85-57e298b7604a" containerID="e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e" exitCode=0
Sep 17 16:31:24.194644 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.194609    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"
Sep 17 16:31:24.194644 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.194622    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" event={"ID":"16512504-2307-4f7d-9b85-57e298b7604a","Type":"ContainerDied","Data":"02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26"}
Sep 17 16:31:24.194974 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.194674    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" event={"ID":"16512504-2307-4f7d-9b85-57e298b7604a","Type":"ContainerDied","Data":"e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e"}
Sep 17 16:31:24.194974 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.194693    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j" event={"ID":"16512504-2307-4f7d-9b85-57e298b7604a","Type":"ContainerDied","Data":"aec1da052d1e8ca9b01c8845c8dfbdaba6d6faa67633c1e2a9503e9d3a82f4d8"}
Sep 17 16:31:24.194974 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.194713    2555 scope.go:117] "RemoveContainer" containerID="02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26"
Sep 17 16:31:24.204080 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.204061    2555 scope.go:117] "RemoveContainer" containerID="8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e"
Sep 17 16:31:24.212246 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.212219    2555 scope.go:117] "RemoveContainer" containerID="e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e"
Sep 17 16:31:24.220626 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.220422    2555 scope.go:117] "RemoveContainer" containerID="02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26"
Sep 17 16:31:24.220933 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:31:24.220898    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26\": container with ID starting with 02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26 not found: ID does not exist" containerID="02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26"
Sep 17 16:31:24.221058 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.220949    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26"} err="failed to get container status \"02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26\": rpc error: code = NotFound desc = could not find container \"02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26\": container with ID starting with 02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26 not found: ID does not exist"
Sep 17 16:31:24.221058 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.220979    2555 scope.go:117] "RemoveContainer" containerID="8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e"
Sep 17 16:31:24.221380 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:31:24.221354    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e\": container with ID starting with 8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e not found: ID does not exist" containerID="8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e"
Sep 17 16:31:24.221480 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.221386    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e"} err="failed to get container status \"8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e\": rpc error: code = NotFound desc = could not find container \"8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e\": container with ID starting with 8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e not found: ID does not exist"
Sep 17 16:31:24.221480 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.221411    2555 scope.go:117] "RemoveContainer" containerID="e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e"
Sep 17 16:31:24.221769 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:31:24.221753    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e\": container with ID starting with e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e not found: ID does not exist" containerID="e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e"
Sep 17 16:31:24.221813 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.221773    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e"} err="failed to get container status \"e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e\": rpc error: code = NotFound desc = could not find container \"e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e\": container with ID starting with e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e not found: ID does not exist"
Sep 17 16:31:24.221813 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.221797    2555 scope.go:117] "RemoveContainer" containerID="02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26"
Sep 17 16:31:24.222150 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.222129    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26"} err="failed to get container status \"02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26\": rpc error: code = NotFound desc = could not find container \"02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26\": container with ID starting with 02f1fd69873f45bc1fb4cad90c5fcc137f5e22e73e962d4d9b678d9587706d26 not found: ID does not exist"
Sep 17 16:31:24.222214 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.222151    2555 scope.go:117] "RemoveContainer" containerID="8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e"
Sep 17 16:31:24.222417 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.222400    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e"} err="failed to get container status \"8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e\": rpc error: code = NotFound desc = could not find container \"8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e\": container with ID starting with 8ebe00a2da49a6d275128237b98cd206e3a3889cf23de1a310eb3f26b39d895e not found: ID does not exist"
Sep 17 16:31:24.222460 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.222417    2555 scope.go:117] "RemoveContainer" containerID="e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e"
Sep 17 16:31:24.222667 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.222650    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e"} err="failed to get container status \"e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e\": rpc error: code = NotFound desc = could not find container \"e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e\": container with ID starting with e4395331966cca217073f3df15fffde94572cc95467af9e5e1e38ed8ea430b7e not found: ID does not exist"
Sep 17 16:31:24.223539 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.223518    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"]
Sep 17 16:31:24.229707 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.229680    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c7ff48d9-zvp6j"]
Sep 17 16:31:24.694034 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:24.694000    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16512504-2307-4f7d-9b85-57e298b7604a" path="/var/lib/kubelet/pods/16512504-2307-4f7d-9b85-57e298b7604a/volumes"
Sep 17 16:31:58.360976 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.360940    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"]
Sep 17 16:31:58.361521 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.361403    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main"
Sep 17 16:31:58.361521 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.361424    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main"
Sep 17 16:31:58.361521 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.361438    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="llm-d-routing-sidecar"
Sep 17 16:31:58.361521 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.361448    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="llm-d-routing-sidecar"
Sep 17 16:31:58.361521 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.361463    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="storage-initializer"
Sep 17 16:31:58.361521 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.361474    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="storage-initializer"
Sep 17 16:31:58.361867 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.361562    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="llm-d-routing-sidecar"
Sep 17 16:31:58.361867 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.361578    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="16512504-2307-4f7d-9b85-57e298b7604a" containerName="main"
Sep 17 16:31:58.364337 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.364313    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.370414 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.370373    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 16:31:58.370414 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.370415    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 16:31:58.370722 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.370593    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-f9bmn\""
Sep 17 16:31:58.370722 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.370704    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 16:31:58.453845 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.453777    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"]
Sep 17 16:31:58.505409 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.505370    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742sg\" (UniqueName: \"kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-kube-api-access-742sg\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.505574 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.505419    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.505574 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.505482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.505574 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.505523    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.505574 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.505545    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.505574 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.505571    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba390ea7-8912-4644-8147-c8041bddd874-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.505781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.505631    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.505781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.505666    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba390ea7-8912-4644-8147-c8041bddd874-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.505781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.505688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.606298 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606202    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.606298 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.606298 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606270    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.606599 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606303    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.606599 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606340    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba390ea7-8912-4644-8147-c8041bddd874-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.606599 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606394    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.606599 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606432    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba390ea7-8912-4644-8147-c8041bddd874-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.606599 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606458    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.606599 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606491    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742sg\" (UniqueName: \"kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-kube-api-access-742sg\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.607041 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606682    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.607041 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606721    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.607041 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606933    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.607041 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.606966    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.607315 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.607285    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba390ea7-8912-4644-8147-c8041bddd874-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.609296 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.609273    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.609555 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.609535    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba390ea7-8912-4644-8147-c8041bddd874-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.616170 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.616149    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.616383 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.616358    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-742sg\" (UniqueName: \"kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-kube-api-access-742sg\") pod \"router-gateway-2-openshift-default-844b774c6-8nzpf\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.674873 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.674805    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:31:58.960155 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.960128    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"]
Sep 17 16:31:58.962418 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:31:58.962389    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba390ea7_8912_4644_8147_c8041bddd874.slice/crio-8c8ffb8c5c055870cd1b87e67863a5da580a678f1fc9486b645804117a041d97 WatchSource:0}: Error finding container 8c8ffb8c5c055870cd1b87e67863a5da580a678f1fc9486b645804117a041d97: Status 404 returned error can't find the container with id 8c8ffb8c5c055870cd1b87e67863a5da580a678f1fc9486b645804117a041d97
Sep 17 16:31:58.964502 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.964469    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:31:58.964609 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.964536    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:31:58.964609 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:58.964569    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:31:59.364233 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:59.364194    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" event={"ID":"ba390ea7-8912-4644-8147-c8041bddd874","Type":"ContainerStarted","Data":"593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d"}
Sep 17 16:31:59.364779 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:59.364261    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" event={"ID":"ba390ea7-8912-4644-8147-c8041bddd874","Type":"ContainerStarted","Data":"8c8ffb8c5c055870cd1b87e67863a5da580a678f1fc9486b645804117a041d97"}
Sep 17 16:31:59.397351 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:59.397294    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" podStartSLOduration=1.3972752050000001 podStartE2EDuration="1.397275205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:31:59.395593847 +0000 UTC m=+3201.286454390" watchObservedRunningTime="2026-09-17 16:31:59.397275205 +0000 UTC m=+3201.288135775"
Sep 17 16:31:59.675309 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:31:59.675199    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:32:00.681195 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:00.681160    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:32:01.374203 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:01.374160    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:32:01.375409 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:01.375374    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:32:07.549563 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.549466    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"]
Sep 17 16:32:07.553118 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.553094    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.556064 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.556033    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 16:32:07.556462 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.556434    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-zjh7q\""
Sep 17 16:32:07.556605 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.556435    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-m72m4\""
Sep 17 16:32:07.556763 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.556741    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 16:32:07.556995 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.556969    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:32:07.590759 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.590710    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"]
Sep 17 16:32:07.690100 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.690051    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-home\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.690335 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.690129    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-tmp-dir\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.690335 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.690154    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-dshm\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.690335 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.690170    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea70259e-7d4d-4484-8f3f-472046835efa-tls-certs\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.690335 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.690206    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-model-cache\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.690335 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.690270    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea70259e-7d4d-4484-8f3f-472046835efa-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.690335 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.690308    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wzk9\" (UniqueName: \"kubernetes.io/projected/ea70259e-7d4d-4484-8f3f-472046835efa-kube-api-access-9wzk9\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.690680 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.690342    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.791608 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.791553    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-home\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.791810 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.791719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-tmp-dir\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.791810 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.791759    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-dshm\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.791810 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.791785    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea70259e-7d4d-4484-8f3f-472046835efa-tls-certs\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.792045 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.791852    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-model-cache\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.792045 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.791921    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea70259e-7d4d-4484-8f3f-472046835efa-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.792045 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.791950    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wzk9\" (UniqueName: \"kubernetes.io/projected/ea70259e-7d4d-4484-8f3f-472046835efa-kube-api-access-9wzk9\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.792045 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.791990    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.792251 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.792091    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-home\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.792322 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.792297    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-model-cache\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.792459 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.792417    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-tmp-dir\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.792596 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.792482    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.792863 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.792844    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea70259e-7d4d-4484-8f3f-472046835efa-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.794899 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.794873    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-dshm\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.795111 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.795096    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea70259e-7d4d-4484-8f3f-472046835efa-tls-certs\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.801042 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.800969    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wzk9\" (UniqueName: \"kubernetes.io/projected/ea70259e-7d4d-4484-8f3f-472046835efa-kube-api-access-9wzk9\") pod \"router-with-refs-pd-test-kserve-77f48b5b59-m9bz7\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:07.866467 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:07.866420    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:08.156545 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:08.156517    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"]
Sep 17 16:32:08.158491 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:32:08.158452    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea70259e_7d4d_4484_8f3f_472046835efa.slice/crio-4a37ef183614d3b9fbd2a097b8b88c15b2da7923fbf8e2ccaf1ee2946f9421e1 WatchSource:0}: Error finding container 4a37ef183614d3b9fbd2a097b8b88c15b2da7923fbf8e2ccaf1ee2946f9421e1: Status 404 returned error can't find the container with id 4a37ef183614d3b9fbd2a097b8b88c15b2da7923fbf8e2ccaf1ee2946f9421e1
Sep 17 16:32:08.417026 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:08.416911    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" event={"ID":"ea70259e-7d4d-4484-8f3f-472046835efa","Type":"ContainerStarted","Data":"79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605"}
Sep 17 16:32:08.417026 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:08.417000    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" event={"ID":"ea70259e-7d4d-4484-8f3f-472046835efa","Type":"ContainerStarted","Data":"4a37ef183614d3b9fbd2a097b8b88c15b2da7923fbf8e2ccaf1ee2946f9421e1"}
Sep 17 16:32:08.417224 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:08.417038    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:09.425286 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:09.425245    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" event={"ID":"ea70259e-7d4d-4484-8f3f-472046835efa","Type":"ContainerStarted","Data":"c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8"}
Sep 17 16:32:10.431544 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:10.431503    2555 generic.go:356] "Generic (PLEG): container finished" podID="ba390ea7-8912-4644-8147-c8041bddd874" containerID="593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d" exitCode=0
Sep 17 16:32:10.432056 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:10.431586    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" event={"ID":"ba390ea7-8912-4644-8147-c8041bddd874","Type":"ContainerDied","Data":"593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d"}
Sep 17 16:32:10.432240 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:10.432211    2555 scope.go:117] "RemoveContainer" containerID="593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d"
Sep 17 16:32:10.433301 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:10.433272    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:32:10.433364 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:10.433340    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:32:10.433400 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:10.433375    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 16:32:11.443382 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:11.443335    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" event={"ID":"ba390ea7-8912-4644-8147-c8041bddd874","Type":"ContainerStarted","Data":"3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f"}
Sep 17 16:32:11.446449 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:11.446405    2555 generic.go:356] "Generic (PLEG): container finished" podID="ea70259e-7d4d-4484-8f3f-472046835efa" containerID="c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8" exitCode=0
Sep 17 16:32:11.446672 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:11.446558    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" event={"ID":"ea70259e-7d4d-4484-8f3f-472046835efa","Type":"ContainerDied","Data":"c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8"}
Sep 17 16:32:11.676051 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:11.676008    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:32:11.677263 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:11.677231    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 17 16:32:12.458450 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:12.458408    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" event={"ID":"ea70259e-7d4d-4484-8f3f-472046835efa","Type":"ContainerStarted","Data":"297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545"}
Sep 17 16:32:12.488851 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:12.488760    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podStartSLOduration=2.955851098 podStartE2EDuration="5.488721304s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.532870206s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:32:12.485037489 +0000 UTC m=+3214.375898040" watchObservedRunningTime="2026-09-17 16:32:12.488721304 +0000 UTC m=+3214.379581849"
Sep 17 16:32:12.676234 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:12.676185    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 17 16:32:13.676233 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:13.676189    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 17 16:32:14.675645 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:14.675601    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 17 16:32:15.676200 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:15.676155    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 17 16:32:16.374167 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:16.374118    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:32:16.680986 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:16.680944    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:32:16.682422 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:16.682393    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:32:17.866981 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:17.866938    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:17.867412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:17.867001    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:17.868307 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:17.868279    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:32:27.867433 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:27.867371    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:32:27.880189 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:27.880156    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:32:37.867457 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:37.867320    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:32:47.867155 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:47.867092    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:32:57.867026 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:32:57.866979    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:33:07.867552 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:33:07.867489    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:33:17.867255 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:33:17.867204    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:33:27.867783 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:33:27.867725    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:33:37.867387 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:33:37.867329    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:33:38.958642 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:33:38.958613    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:33:38.963560 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:33:38.963528    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:33:47.867142 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:33:47.867093    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:33:57.866955 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:33:57.866898    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:34:07.867036 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:07.866989    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 17 16:34:17.884190 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:17.884157    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:34:17.896182 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:17.896146    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:34:27.675354 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:27.675317    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"]
Sep 17 16:34:42.699206 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:42.699136    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main" containerID="cri-o://297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545" gracePeriod=45
Sep 17 16:34:43.316860 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.316809    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="llm-d-routing-sidecar" containerID="cri-o://79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605" gracePeriod=45
Sep 17 16:34:43.563961 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.563933    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:34:43.661022 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.660983    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wzk9\" (UniqueName: \"kubernetes.io/projected/ea70259e-7d4d-4484-8f3f-472046835efa-kube-api-access-9wzk9\") pod \"ea70259e-7d4d-4484-8f3f-472046835efa\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") "
Sep 17 16:34:43.661224 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.661042    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-home\") pod \"ea70259e-7d4d-4484-8f3f-472046835efa\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") "
Sep 17 16:34:43.661224 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.661059    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-tmp-dir\") pod \"ea70259e-7d4d-4484-8f3f-472046835efa\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") "
Sep 17 16:34:43.661224 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.661089    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea70259e-7d4d-4484-8f3f-472046835efa-cabundle-cert\") pod \"ea70259e-7d4d-4484-8f3f-472046835efa\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") "
Sep 17 16:34:43.661224 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.661112    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-dshm\") pod \"ea70259e-7d4d-4484-8f3f-472046835efa\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") "
Sep 17 16:34:43.661224 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.661132    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-kserve-provision-location\") pod \"ea70259e-7d4d-4484-8f3f-472046835efa\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") "
Sep 17 16:34:43.661224 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.661156    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea70259e-7d4d-4484-8f3f-472046835efa-tls-certs\") pod \"ea70259e-7d4d-4484-8f3f-472046835efa\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") "
Sep 17 16:34:43.661224 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.661177    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-model-cache\") pod \"ea70259e-7d4d-4484-8f3f-472046835efa\" (UID: \"ea70259e-7d4d-4484-8f3f-472046835efa\") "
Sep 17 16:34:43.661628 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.661467    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea70259e-7d4d-4484-8f3f-472046835efa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea70259e-7d4d-4484-8f3f-472046835efa" (UID: "ea70259e-7d4d-4484-8f3f-472046835efa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:43.661628 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.661608    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea70259e-7d4d-4484-8f3f-472046835efa" (UID: "ea70259e-7d4d-4484-8f3f-472046835efa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:43.661806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.661782    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-home" (OuterVolumeSpecName: "home") pod "ea70259e-7d4d-4484-8f3f-472046835efa" (UID: "ea70259e-7d4d-4484-8f3f-472046835efa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:43.663650 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.663616    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-dshm" (OuterVolumeSpecName: "dshm") pod "ea70259e-7d4d-4484-8f3f-472046835efa" (UID: "ea70259e-7d4d-4484-8f3f-472046835efa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:43.663775 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.663650    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea70259e-7d4d-4484-8f3f-472046835efa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea70259e-7d4d-4484-8f3f-472046835efa" (UID: "ea70259e-7d4d-4484-8f3f-472046835efa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:34:43.664006 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.663865    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea70259e-7d4d-4484-8f3f-472046835efa-kube-api-access-9wzk9" (OuterVolumeSpecName: "kube-api-access-9wzk9") pod "ea70259e-7d4d-4484-8f3f-472046835efa" (UID: "ea70259e-7d4d-4484-8f3f-472046835efa"). InnerVolumeSpecName "kube-api-access-9wzk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:34:43.674367 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.674324    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea70259e-7d4d-4484-8f3f-472046835efa" (UID: "ea70259e-7d4d-4484-8f3f-472046835efa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:43.685661 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.685622    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea70259e-7d4d-4484-8f3f-472046835efa" (UID: "ea70259e-7d4d-4484-8f3f-472046835efa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:43.762117 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.762075    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea70259e-7d4d-4484-8f3f-472046835efa-cabundle-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.762117 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.762110    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.762117 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.762120    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-kserve-provision-location\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.762598 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.762135    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea70259e-7d4d-4484-8f3f-472046835efa-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.762598 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.762146    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.762598 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.762154    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wzk9\" (UniqueName: \"kubernetes.io/projected/ea70259e-7d4d-4484-8f3f-472046835efa-kube-api-access-9wzk9\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.762598 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.762162    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.762598 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:43.762171    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea70259e-7d4d-4484-8f3f-472046835efa-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:34:44.279772 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.279728    2555 generic.go:356] "Generic (PLEG): container finished" podID="ea70259e-7d4d-4484-8f3f-472046835efa" containerID="297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545" exitCode=0
Sep 17 16:34:44.279772 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.279764    2555 generic.go:356] "Generic (PLEG): container finished" podID="ea70259e-7d4d-4484-8f3f-472046835efa" containerID="79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605" exitCode=0
Sep 17 16:34:44.280024 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.279811    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"
Sep 17 16:34:44.280024 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.279810    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" event={"ID":"ea70259e-7d4d-4484-8f3f-472046835efa","Type":"ContainerDied","Data":"297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545"}
Sep 17 16:34:44.280024 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.279890    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" event={"ID":"ea70259e-7d4d-4484-8f3f-472046835efa","Type":"ContainerDied","Data":"79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605"}
Sep 17 16:34:44.280024 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.279909    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7" event={"ID":"ea70259e-7d4d-4484-8f3f-472046835efa","Type":"ContainerDied","Data":"4a37ef183614d3b9fbd2a097b8b88c15b2da7923fbf8e2ccaf1ee2946f9421e1"}
Sep 17 16:34:44.280024 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.279930    2555 scope.go:117] "RemoveContainer" containerID="297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545"
Sep 17 16:34:44.292005 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.291982    2555 scope.go:117] "RemoveContainer" containerID="c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8"
Sep 17 16:34:44.301570 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.301545    2555 scope.go:117] "RemoveContainer" containerID="79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605"
Sep 17 16:34:44.309949 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.309925    2555 scope.go:117] "RemoveContainer" containerID="297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545"
Sep 17 16:34:44.310290 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:34:44.310269    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545\": container with ID starting with 297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545 not found: ID does not exist" containerID="297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545"
Sep 17 16:34:44.310342 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.310303    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545"} err="failed to get container status \"297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545\": rpc error: code = NotFound desc = could not find container \"297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545\": container with ID starting with 297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545 not found: ID does not exist"
Sep 17 16:34:44.310342 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.310324    2555 scope.go:117] "RemoveContainer" containerID="c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8"
Sep 17 16:34:44.310634 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:34:44.310614    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8\": container with ID starting with c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8 not found: ID does not exist" containerID="c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8"
Sep 17 16:34:44.310677 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.310646    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8"} err="failed to get container status \"c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8\": rpc error: code = NotFound desc = could not find container \"c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8\": container with ID starting with c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8 not found: ID does not exist"
Sep 17 16:34:44.310677 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.310663    2555 scope.go:117] "RemoveContainer" containerID="79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605"
Sep 17 16:34:44.310937 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:34:44.310916    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605\": container with ID starting with 79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605 not found: ID does not exist" containerID="79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605"
Sep 17 16:34:44.311009 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.310941    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605"} err="failed to get container status \"79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605\": rpc error: code = NotFound desc = could not find container \"79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605\": container with ID starting with 79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605 not found: ID does not exist"
Sep 17 16:34:44.311009 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.310955    2555 scope.go:117] "RemoveContainer" containerID="297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545"
Sep 17 16:34:44.311188 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.311171    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545"} err="failed to get container status \"297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545\": rpc error: code = NotFound desc = could not find container \"297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545\": container with ID starting with 297ce0fbb2552d84f8e7ad145f648e6badd6860d837c56d42a006fb4bba9e545 not found: ID does not exist"
Sep 17 16:34:44.311230 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.311190    2555 scope.go:117] "RemoveContainer" containerID="c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8"
Sep 17 16:34:44.311422 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.311406    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8"} err="failed to get container status \"c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8\": rpc error: code = NotFound desc = could not find container \"c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8\": container with ID starting with c68293775190a76326bfa20dbbe371e614c233c795942b0891d6167e115d6de8 not found: ID does not exist"
Sep 17 16:34:44.311479 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.311422    2555 scope.go:117] "RemoveContainer" containerID="79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605"
Sep 17 16:34:44.311660 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.311640    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605"} err="failed to get container status \"79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605\": rpc error: code = NotFound desc = could not find container \"79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605\": container with ID starting with 79ccfcdb320ea8bbbe27169ed9530a8fae08527e1215e33bd8076ed4561d3605 not found: ID does not exist"
Sep 17 16:34:44.321936 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.321913    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"]
Sep 17 16:34:44.327268 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.327240    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-77f48b5b59-m9bz7"]
Sep 17 16:34:44.694112 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:34:44.694077    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" path="/var/lib/kubelet/pods/ea70259e-7d4d-4484-8f3f-472046835efa/volumes"
Sep 17 16:35:22.769973 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:22.769936    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"]
Sep 17 16:35:22.770510 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:22.770142    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy" containerID="cri-o://3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f" gracePeriod=30
Sep 17 16:35:28.026083 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.026051    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:35:28.127153 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127116    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-742sg\" (UniqueName: \"kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-kube-api-access-742sg\") pod \"ba390ea7-8912-4644-8147-c8041bddd874\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") "
Sep 17 16:35:28.127153 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127161    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-credential-socket\") pod \"ba390ea7-8912-4644-8147-c8041bddd874\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") "
Sep 17 16:35:28.127405 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127179    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba390ea7-8912-4644-8147-c8041bddd874-istiod-ca-cert\") pod \"ba390ea7-8912-4644-8147-c8041bddd874\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") "
Sep 17 16:35:28.127405 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127217    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-certs\") pod \"ba390ea7-8912-4644-8147-c8041bddd874\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") "
Sep 17 16:35:28.127405 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127234    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-data\") pod \"ba390ea7-8912-4644-8147-c8041bddd874\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") "
Sep 17 16:35:28.127405 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127251    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-socket\") pod \"ba390ea7-8912-4644-8147-c8041bddd874\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") "
Sep 17 16:35:28.127405 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127300    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba390ea7-8912-4644-8147-c8041bddd874-istio-podinfo\") pod \"ba390ea7-8912-4644-8147-c8041bddd874\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") "
Sep 17 16:35:28.127405 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127340    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-istio-token\") pod \"ba390ea7-8912-4644-8147-c8041bddd874\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") "
Sep 17 16:35:28.127405 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127358    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-envoy\") pod \"ba390ea7-8912-4644-8147-c8041bddd874\" (UID: \"ba390ea7-8912-4644-8147-c8041bddd874\") "
Sep 17 16:35:28.127774 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127448    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba390ea7-8912-4644-8147-c8041bddd874" (UID: "ba390ea7-8912-4644-8147-c8041bddd874"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:35:28.127774 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127598    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-credential-socket\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:35:28.127900 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127777    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba390ea7-8912-4644-8147-c8041bddd874" (UID: "ba390ea7-8912-4644-8147-c8041bddd874"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:35:28.127900 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127788    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba390ea7-8912-4644-8147-c8041bddd874" (UID: "ba390ea7-8912-4644-8147-c8041bddd874"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:35:28.127900 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127851    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba390ea7-8912-4644-8147-c8041bddd874-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba390ea7-8912-4644-8147-c8041bddd874" (UID: "ba390ea7-8912-4644-8147-c8041bddd874"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:35:28.127900 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.127872    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba390ea7-8912-4644-8147-c8041bddd874" (UID: "ba390ea7-8912-4644-8147-c8041bddd874"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:35:28.129576 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.129535    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba390ea7-8912-4644-8147-c8041bddd874" (UID: "ba390ea7-8912-4644-8147-c8041bddd874"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:35:28.129576 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.129564    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba390ea7-8912-4644-8147-c8041bddd874" (UID: "ba390ea7-8912-4644-8147-c8041bddd874"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:35:28.129815 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.129799    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-kube-api-access-742sg" (OuterVolumeSpecName: "kube-api-access-742sg") pod "ba390ea7-8912-4644-8147-c8041bddd874" (UID: "ba390ea7-8912-4644-8147-c8041bddd874"). InnerVolumeSpecName "kube-api-access-742sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:35:28.129815 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.129806    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba390ea7-8912-4644-8147-c8041bddd874-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba390ea7-8912-4644-8147-c8041bddd874" (UID: "ba390ea7-8912-4644-8147-c8041bddd874"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 16:35:28.228526 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.228486    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba390ea7-8912-4644-8147-c8041bddd874-istio-podinfo\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:35:28.228526 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.228519    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-istio-token\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:35:28.228526 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.228532    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-envoy\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:35:28.228782 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.228545    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-742sg\" (UniqueName: \"kubernetes.io/projected/ba390ea7-8912-4644-8147-c8041bddd874-kube-api-access-742sg\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:35:28.228782 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.228557    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba390ea7-8912-4644-8147-c8041bddd874-istiod-ca-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:35:28.228782 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.228568    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:35:28.228782 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.228578    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-istio-data\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:35:28.228782 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.228588    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba390ea7-8912-4644-8147-c8041bddd874-workload-socket\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:35:28.504699 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.504612    2555 generic.go:356] "Generic (PLEG): container finished" podID="ba390ea7-8912-4644-8147-c8041bddd874" containerID="3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f" exitCode=0
Sep 17 16:35:28.504699 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.504662    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"
Sep 17 16:35:28.504699 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.504682    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" event={"ID":"ba390ea7-8912-4644-8147-c8041bddd874","Type":"ContainerDied","Data":"3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f"}
Sep 17 16:35:28.504950 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.504725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf" event={"ID":"ba390ea7-8912-4644-8147-c8041bddd874","Type":"ContainerDied","Data":"8c8ffb8c5c055870cd1b87e67863a5da580a678f1fc9486b645804117a041d97"}
Sep 17 16:35:28.504950 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.504747    2555 scope.go:117] "RemoveContainer" containerID="3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f"
Sep 17 16:35:28.514060 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.514043    2555 scope.go:117] "RemoveContainer" containerID="593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d"
Sep 17 16:35:28.521665 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.521649    2555 scope.go:117] "RemoveContainer" containerID="3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f"
Sep 17 16:35:28.521937 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:35:28.521918    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f\": container with ID starting with 3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f not found: ID does not exist" containerID="3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f"
Sep 17 16:35:28.522017 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.521949    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f"} err="failed to get container status \"3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f\": rpc error: code = NotFound desc = could not find container \"3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f\": container with ID starting with 3daf6931556e7f71010f96682f792ba324db5af27956dac8d81969d33f792e0f not found: ID does not exist"
Sep 17 16:35:28.522017 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.521973    2555 scope.go:117] "RemoveContainer" containerID="593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d"
Sep 17 16:35:28.522245 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:35:28.522227    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d\": container with ID starting with 593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d not found: ID does not exist" containerID="593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d"
Sep 17 16:35:28.522288 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.522251    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d"} err="failed to get container status \"593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d\": rpc error: code = NotFound desc = could not find container \"593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d\": container with ID starting with 593b31684a4f5e0e6cedc65b5316380f95f967d0f6656076dd2845217928cf2d not found: ID does not exist"
Sep 17 16:35:28.533156 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.533129    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"]
Sep 17 16:35:28.539976 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.539954    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-8nzpf"]
Sep 17 16:35:28.693526 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:28.693497    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba390ea7-8912-4644-8147-c8041bddd874" path="/var/lib/kubelet/pods/ba390ea7-8912-4644-8147-c8041bddd874/volumes"
Sep 17 16:35:33.915054 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.914974    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"]
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915247    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915257    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915266    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="storage-initializer"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915271    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="storage-initializer"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915285    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915290    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915296    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915301    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915312    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="llm-d-routing-sidecar"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915317    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="llm-d-routing-sidecar"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915361    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915367    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="llm-d-routing-sidecar"
Sep 17 16:35:33.915412 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915376    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea70259e-7d4d-4484-8f3f-472046835efa" containerName="main"
Sep 17 16:35:33.915795 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.915465    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba390ea7-8912-4644-8147-c8041bddd874" containerName="istio-proxy"
Sep 17 16:35:33.919709 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.919692    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:33.922532 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.922513    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:35:33.923085 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.923067    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 17 16:35:33.923166 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.923150    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-ttrkl\""
Sep 17 16:35:33.923349 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.923335    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 17 16:35:33.923957 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.923938    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 17 16:35:33.924053 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.923968    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 17 16:35:33.952983 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:33.952961    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"]
Sep 17 16:35:34.072399 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.072375    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.072524 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.072404    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.072524 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.072428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.072524 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.072469    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.072524 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.072508    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.072662 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.072533    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.072662 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.072552    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.072662 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.072594    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxrf\" (UniqueName: \"kubernetes.io/projected/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kube-api-access-7wxrf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.173540 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.173475    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.173540 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.173503    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.173540 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.173525    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.173781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.173545    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.173781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.173574    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.173781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.173603    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.173781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.173630    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.173781 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.173679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxrf\" (UniqueName: \"kubernetes.io/projected/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kube-api-access-7wxrf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.174072 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.173969    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.174072 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.173996    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.174160 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.174109    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.174196 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.174165    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.174463 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.174446    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.175956 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.175939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.176229 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.176211    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.200766 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.200735    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxrf\" (UniqueName: \"kubernetes.io/projected/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kube-api-access-7wxrf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.230004 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.229969    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:34.559406 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:34.559380    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"]
Sep 17 16:35:34.559643 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:35:34.559620    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2c07e8_ff54_4dcf_996c_09961d6b1c22.slice/crio-2eca8c6a8091642360883302942094f87ef041297b216b11f708983cf6faed82 WatchSource:0}: Error finding container 2eca8c6a8091642360883302942094f87ef041297b216b11f708983cf6faed82: Status 404 returned error can't find the container with id 2eca8c6a8091642360883302942094f87ef041297b216b11f708983cf6faed82
Sep 17 16:35:35.541834 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:35.541783    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" event={"ID":"fa2c07e8-ff54-4dcf-996c-09961d6b1c22","Type":"ContainerStarted","Data":"bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05"}
Sep 17 16:35:35.542216 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:35.541853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" event={"ID":"fa2c07e8-ff54-4dcf-996c-09961d6b1c22","Type":"ContainerStarted","Data":"2eca8c6a8091642360883302942094f87ef041297b216b11f708983cf6faed82"}
Sep 17 16:35:37.554109 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:37.554074    2555 generic.go:356] "Generic (PLEG): container finished" podID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerID="bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05" exitCode=0
Sep 17 16:35:37.554525 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:37.554146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" event={"ID":"fa2c07e8-ff54-4dcf-996c-09961d6b1c22","Type":"ContainerDied","Data":"bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05"}
Sep 17 16:35:38.560420 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:38.560388    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" event={"ID":"fa2c07e8-ff54-4dcf-996c-09961d6b1c22","Type":"ContainerStarted","Data":"bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794"}
Sep 17 16:35:39.230729 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:39.230698    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:44.230424 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:44.230391    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:35:44.231757 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:44.231729    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:35:54.231473 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:35:54.231425    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:36:04.230538 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:36:04.230493    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:36:14.230655 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:36:14.230595    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:36:24.231266 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:36:24.231212    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:36:34.231427 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:36:34.231383    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:36:44.231111 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:36:44.231064    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:36:54.230759 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:36:54.230710    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:37:04.230637 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:04.230542    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:37:14.230926 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:14.230883    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:37:24.240111 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:24.240080    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:37:24.247691 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:24.247669    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:37:24.278667 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:24.278620    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podStartSLOduration=109.006848622 podStartE2EDuration="1m51.278603659s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.271755037s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:35:38.59249574 +0000 UTC m=+3420.483356284" watchObservedRunningTime="2026-09-17 16:37:24.278603659 +0000 UTC m=+3526.169464204"
Sep 17 16:37:28.939535 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:28.939499    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"]
Sep 17 16:37:43.961392 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:43.961333    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" containerID="cri-o://bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794" gracePeriod=45
Sep 17 16:37:44.240975 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.240886    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 16:37:44.614855 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.614816    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:37:44.715892 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.715865    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-home\") pod \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") "
Sep 17 16:37:44.716025 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.715905    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wxrf\" (UniqueName: \"kubernetes.io/projected/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kube-api-access-7wxrf\") pod \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") "
Sep 17 16:37:44.716025 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.715954    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-dshm\") pod \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") "
Sep 17 16:37:44.716025 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.715978    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tmp-dir\") pod \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") "
Sep 17 16:37:44.716025 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.716015    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kserve-provision-location\") pod \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") "
Sep 17 16:37:44.716244 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.716043    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tls-certs\") pod \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") "
Sep 17 16:37:44.716244 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.716068    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-cabundle-cert\") pod \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") "
Sep 17 16:37:44.716244 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.716123    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-model-cache\") pod \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\" (UID: \"fa2c07e8-ff54-4dcf-996c-09961d6b1c22\") "
Sep 17 16:37:44.716600 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.716575    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa2c07e8-ff54-4dcf-996c-09961d6b1c22" (UID: "fa2c07e8-ff54-4dcf-996c-09961d6b1c22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:37:44.716754 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.716596    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-home" (OuterVolumeSpecName: "home") pod "fa2c07e8-ff54-4dcf-996c-09961d6b1c22" (UID: "fa2c07e8-ff54-4dcf-996c-09961d6b1c22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:37:44.716902 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.716735    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa2c07e8-ff54-4dcf-996c-09961d6b1c22" (UID: "fa2c07e8-ff54-4dcf-996c-09961d6b1c22"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:37:44.718309 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.718283    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-dshm" (OuterVolumeSpecName: "dshm") pod "fa2c07e8-ff54-4dcf-996c-09961d6b1c22" (UID: "fa2c07e8-ff54-4dcf-996c-09961d6b1c22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:37:44.718396 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.718333    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa2c07e8-ff54-4dcf-996c-09961d6b1c22" (UID: "fa2c07e8-ff54-4dcf-996c-09961d6b1c22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:37:44.718444 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.718435    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kube-api-access-7wxrf" (OuterVolumeSpecName: "kube-api-access-7wxrf") pod "fa2c07e8-ff54-4dcf-996c-09961d6b1c22" (UID: "fa2c07e8-ff54-4dcf-996c-09961d6b1c22"). InnerVolumeSpecName "kube-api-access-7wxrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:37:44.734136 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.734111    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa2c07e8-ff54-4dcf-996c-09961d6b1c22" (UID: "fa2c07e8-ff54-4dcf-996c-09961d6b1c22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:37:44.739988 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.739966    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa2c07e8-ff54-4dcf-996c-09961d6b1c22" (UID: "fa2c07e8-ff54-4dcf-996c-09961d6b1c22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:37:44.817184 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.817131    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:37:44.817184 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.817157    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wxrf\" (UniqueName: \"kubernetes.io/projected/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kube-api-access-7wxrf\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:37:44.817184 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.817172    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:37:44.817184 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.817185    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:37:44.817371 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.817198    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-kserve-provision-location\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:37:44.817371 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.817210    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:37:44.817371 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.817222    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-cabundle-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:37:44.817371 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:44.817233    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2c07e8-ff54-4dcf-996c-09961d6b1c22-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:37:45.189144 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.189111    2555 generic.go:356] "Generic (PLEG): container finished" podID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerID="bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794" exitCode=0
Sep 17 16:37:45.189528 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.189167    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"
Sep 17 16:37:45.189528 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.189195    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" event={"ID":"fa2c07e8-ff54-4dcf-996c-09961d6b1c22","Type":"ContainerDied","Data":"bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794"}
Sep 17 16:37:45.189528 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.189247    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6" event={"ID":"fa2c07e8-ff54-4dcf-996c-09961d6b1c22","Type":"ContainerDied","Data":"2eca8c6a8091642360883302942094f87ef041297b216b11f708983cf6faed82"}
Sep 17 16:37:45.189528 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.189270    2555 scope.go:117] "RemoveContainer" containerID="bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794"
Sep 17 16:37:45.199534 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.199517    2555 scope.go:117] "RemoveContainer" containerID="bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05"
Sep 17 16:37:45.207618 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.207602    2555 scope.go:117] "RemoveContainer" containerID="bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794"
Sep 17 16:37:45.207925 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:37:45.207903    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794\": container with ID starting with bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794 not found: ID does not exist" containerID="bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794"
Sep 17 16:37:45.207989 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.207933    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794"} err="failed to get container status \"bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794\": rpc error: code = NotFound desc = could not find container \"bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794\": container with ID starting with bd833dc38b39f31b20908a4fb64b2071d2fa4d577c5d7cb6cfb486e37f9be794 not found: ID does not exist"
Sep 17 16:37:45.207989 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.207948    2555 scope.go:117] "RemoveContainer" containerID="bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05"
Sep 17 16:37:45.208213 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:37:45.208194    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05\": container with ID starting with bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05 not found: ID does not exist" containerID="bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05"
Sep 17 16:37:45.208269 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.208224    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05"} err="failed to get container status \"bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05\": rpc error: code = NotFound desc = could not find container \"bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05\": container with ID starting with bb0fd5c67c3730f10fcb4719fd22076b884d9bcc218d005cfeaafc189b730e05 not found: ID does not exist"
Sep 17 16:37:45.219382 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.219360    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"]
Sep 17 16:37:45.221102 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:45.221083    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69df8ccc6-hq9t6"]
Sep 17 16:37:46.700397 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:46.700359    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" path="/var/lib/kubelet/pods/fa2c07e8-ff54-4dcf-996c-09961d6b1c22/volumes"
Sep 17 16:37:57.812338 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.812309    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 16:37:57.812731 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.812610    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="storage-initializer"
Sep 17 16:37:57.812731 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.812622    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="storage-initializer"
Sep 17 16:37:57.812731 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.812630    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main"
Sep 17 16:37:57.812731 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.812636    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main"
Sep 17 16:37:57.812731 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.812689    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa2c07e8-ff54-4dcf-996c-09961d6b1c22" containerName="main"
Sep 17 16:37:57.815360 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.815344    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:57.824241 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.824217    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 16:37:57.825016 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.824991    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:37:57.825016 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.825008    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 16:37:57.825216 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.825010    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 16:37:57.825216 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.824994    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-dcl7n\""
Sep 17 16:37:57.825389 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.825370    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-cqc55\""
Sep 17 16:37:57.825466 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.825437    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 16:37:57.851891 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.851865    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 16:37:57.906249 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.906220    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:57.906394 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.906261    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:57.906394 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.906286    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aece7fc-7653-4788-b914-291f241caafa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:57.906491 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.906397    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aece7fc-7653-4788-b914-291f241caafa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:57.906491 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.906446    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4m9\" (UniqueName: \"kubernetes.io/projected/8aece7fc-7653-4788-b914-291f241caafa-kube-api-access-qg4m9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:57.906573 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.906522    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:57.906573 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.906561    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:57.906649 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:57.906592    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007425 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007393    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007592 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007440    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007592 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007458    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007592 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007490    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aece7fc-7653-4788-b914-291f241caafa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007592 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007511    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aece7fc-7653-4788-b914-291f241caafa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007592 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4m9\" (UniqueName: \"kubernetes.io/projected/8aece7fc-7653-4788-b914-291f241caafa-kube-api-access-qg4m9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007592 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007582    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007954 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007608    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007954 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007792    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007954 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007849    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.007954 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007946    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.008125 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.007966    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.008125 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.008116    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aece7fc-7653-4788-b914-291f241caafa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.009819 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.009794    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.010023 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.010005    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aece7fc-7653-4788-b914-291f241caafa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.016775 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.016754    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4m9\" (UniqueName: \"kubernetes.io/projected/8aece7fc-7653-4788-b914-291f241caafa-kube-api-access-qg4m9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.127254 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.127231    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:37:58.402841 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:58.402793    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 16:37:58.403754 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:37:58.403727    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aece7fc_7653_4788_b914_291f241caafa.slice/crio-1319c31ce8b150ba27ac9feda3dc8f18a61aa9968c66206d6d0870e821bdb4ae WatchSource:0}: Error finding container 1319c31ce8b150ba27ac9feda3dc8f18a61aa9968c66206d6d0870e821bdb4ae: Status 404 returned error can't find the container with id 1319c31ce8b150ba27ac9feda3dc8f18a61aa9968c66206d6d0870e821bdb4ae
Sep 17 16:37:59.260413 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:59.260376    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8aece7fc-7653-4788-b914-291f241caafa","Type":"ContainerStarted","Data":"6ac5d3993a7f12ca501dd1a7999f1c61deb3aabc8fba70f7f49341a1782a99d2"}
Sep 17 16:37:59.260795 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:37:59.260438    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8aece7fc-7653-4788-b914-291f241caafa","Type":"ContainerStarted","Data":"1319c31ce8b150ba27ac9feda3dc8f18a61aa9968c66206d6d0870e821bdb4ae"}
Sep 17 16:38:01.276406 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:01.276366    2555 generic.go:356] "Generic (PLEG): container finished" podID="8aece7fc-7653-4788-b914-291f241caafa" containerID="6ac5d3993a7f12ca501dd1a7999f1c61deb3aabc8fba70f7f49341a1782a99d2" exitCode=0
Sep 17 16:38:01.276879 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:01.276446    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8aece7fc-7653-4788-b914-291f241caafa","Type":"ContainerDied","Data":"6ac5d3993a7f12ca501dd1a7999f1c61deb3aabc8fba70f7f49341a1782a99d2"}
Sep 17 16:38:02.284058 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:02.284019    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8aece7fc-7653-4788-b914-291f241caafa","Type":"ContainerStarted","Data":"712ba626c9d03ac566e5010594dd96b613110937141e9e34b28a9f3a1820e164"}
Sep 17 16:38:02.326581 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:02.326518    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.057932165 podStartE2EDuration="5.326496858s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.268564693s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:38:02.323071975 +0000 UTC m=+3564.213932518" watchObservedRunningTime="2026-09-17 16:38:02.326496858 +0000 UTC m=+3564.217357403"
Sep 17 16:38:03.128329 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:03.128286    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:38:08.128275 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:08.128241    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:38:08.129439 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:08.129416    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 16:38:18.128503 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:18.128455    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 16:38:28.128324 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:28.128277    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 16:38:38.128280 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:38.128231    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 16:38:38.992463 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:38.992429    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:38:39.000414 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:39.000394    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:38:48.128464 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:48.128414    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 16:38:58.127812 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:38:58.127761    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 16:39:08.128182 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:08.128142    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 16:39:12.999024 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:12.998988    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"]
Sep 17 16:39:13.003316 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.003292    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.010706 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.010679    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 16:39:13.010924 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.010902    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 16:39:13.011081 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.011006    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 16:39:13.011226 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.011010    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-487d4\""
Sep 17 16:39:13.011226 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.011170    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 16:39:13.012156 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.012137    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:39:13.018030 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.018009    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"]
Sep 17 16:39:13.088426 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.088395    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-model-cache\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.088564 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.088428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-kserve-provision-location\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.088564 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.088453    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fca2adf-7f94-4e6f-8f33-e244985da649-cabundle-cert\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.088564 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.088474    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rdxj\" (UniqueName: \"kubernetes.io/projected/6fca2adf-7f94-4e6f-8f33-e244985da649-kube-api-access-8rdxj\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.088564 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.088528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-tmp-dir\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.088564 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.088547    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-home\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.088748 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.088571    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fca2adf-7f94-4e6f-8f33-e244985da649-tls-certs\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.088748 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.088595    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-dshm\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.189449 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.189419    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rdxj\" (UniqueName: \"kubernetes.io/projected/6fca2adf-7f94-4e6f-8f33-e244985da649-kube-api-access-8rdxj\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.189609 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.189457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-tmp-dir\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.189609 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.189478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-home\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.189609 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.189507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fca2adf-7f94-4e6f-8f33-e244985da649-tls-certs\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.189609 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.189538    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-dshm\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.189609 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.189589    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-model-cache\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.189904 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.189612    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-kserve-provision-location\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.189904 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.189650    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fca2adf-7f94-4e6f-8f33-e244985da649-cabundle-cert\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.190019 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.189932    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-tmp-dir\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.190077 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.190021    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-home\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.190117 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.190082    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-model-cache\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.190241 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.190222    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-kserve-provision-location\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.190401 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.190377    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fca2adf-7f94-4e6f-8f33-e244985da649-cabundle-cert\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.191970 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.191946    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-dshm\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.192269 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.192245    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fca2adf-7f94-4e6f-8f33-e244985da649-tls-certs\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.201105 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.201082    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rdxj\" (UniqueName: \"kubernetes.io/projected/6fca2adf-7f94-4e6f-8f33-e244985da649-kube-api-access-8rdxj\") pod \"tls-verification-test-kserve-f4744dcf4-7n2fb\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.322546 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.322472    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:13.607625 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.607592    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"]
Sep 17 16:39:13.609081 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:39:13.609047    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fca2adf_7f94_4e6f_8f33_e244985da649.slice/crio-522929fcc43b4038ccee71352268cea9ad60ad3abafbf3f60c0c83f575a8a204 WatchSource:0}: Error finding container 522929fcc43b4038ccee71352268cea9ad60ad3abafbf3f60c0c83f575a8a204: Status 404 returned error can't find the container with id 522929fcc43b4038ccee71352268cea9ad60ad3abafbf3f60c0c83f575a8a204
Sep 17 16:39:13.646493 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:13.646460    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" event={"ID":"6fca2adf-7f94-4e6f-8f33-e244985da649","Type":"ContainerStarted","Data":"522929fcc43b4038ccee71352268cea9ad60ad3abafbf3f60c0c83f575a8a204"}
Sep 17 16:39:14.656325 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:14.655779    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" event={"ID":"6fca2adf-7f94-4e6f-8f33-e244985da649","Type":"ContainerStarted","Data":"7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f"}
Sep 17 16:39:16.670719 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:16.670680    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerID="7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f" exitCode=0
Sep 17 16:39:16.671150 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:16.670753    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" event={"ID":"6fca2adf-7f94-4e6f-8f33-e244985da649","Type":"ContainerDied","Data":"7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f"}
Sep 17 16:39:17.678736 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:17.678704    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" event={"ID":"6fca2adf-7f94-4e6f-8f33-e244985da649","Type":"ContainerStarted","Data":"f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8"}
Sep 17 16:39:17.705227 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:17.705173    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podStartSLOduration=3.314438827 podStartE2EDuration="5.705156887s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.39071806s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:39:17.702377373 +0000 UTC m=+3639.593237929" watchObservedRunningTime="2026-09-17 16:39:17.705156887 +0000 UTC m=+3639.596017430"
Sep 17 16:39:18.128450 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:18.128402    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 16:39:18.323211 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:18.323176    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:23.323198 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:23.323160    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:39:23.324925 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:23.324897    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:39:28.128172 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:28.128135    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 16:39:33.323710 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:33.323669    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:39:38.138297 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:38.138269    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:39:38.146436 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:38.146411    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:39:43.208925 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:43.208892    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 16:39:43.323150 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:43.323107    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:39:53.322849 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:53.322785    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:39:58.230301 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:58.230254    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main" containerID="cri-o://712ba626c9d03ac566e5010594dd96b613110937141e9e34b28a9f3a1820e164" gracePeriod=45
Sep 17 16:39:58.911387 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:58.911351    2555 generic.go:356] "Generic (PLEG): container finished" podID="8aece7fc-7653-4788-b914-291f241caafa" containerID="712ba626c9d03ac566e5010594dd96b613110937141e9e34b28a9f3a1820e164" exitCode=0
Sep 17 16:39:58.911535 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:58.911425    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8aece7fc-7653-4788-b914-291f241caafa","Type":"ContainerDied","Data":"712ba626c9d03ac566e5010594dd96b613110937141e9e34b28a9f3a1820e164"}
Sep 17 16:39:58.965180 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:58.965157    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:39:59.065209 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065116    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-dshm\") pod \"8aece7fc-7653-4788-b914-291f241caafa\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") "
Sep 17 16:39:59.065209 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065158    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg4m9\" (UniqueName: \"kubernetes.io/projected/8aece7fc-7653-4788-b914-291f241caafa-kube-api-access-qg4m9\") pod \"8aece7fc-7653-4788-b914-291f241caafa\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") "
Sep 17 16:39:59.065209 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065184    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-kserve-provision-location\") pod \"8aece7fc-7653-4788-b914-291f241caafa\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") "
Sep 17 16:39:59.065209 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065209    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-home\") pod \"8aece7fc-7653-4788-b914-291f241caafa\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") "
Sep 17 16:39:59.065550 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065288    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-model-cache\") pod \"8aece7fc-7653-4788-b914-291f241caafa\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") "
Sep 17 16:39:59.065550 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065340    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-tmp-dir\") pod \"8aece7fc-7653-4788-b914-291f241caafa\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") "
Sep 17 16:39:59.065550 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065375    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aece7fc-7653-4788-b914-291f241caafa-cabundle-cert\") pod \"8aece7fc-7653-4788-b914-291f241caafa\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") "
Sep 17 16:39:59.065550 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065405    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aece7fc-7653-4788-b914-291f241caafa-tls-certs\") pod \"8aece7fc-7653-4788-b914-291f241caafa\" (UID: \"8aece7fc-7653-4788-b914-291f241caafa\") "
Sep 17 16:39:59.065772 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065556    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-model-cache" (OuterVolumeSpecName: "model-cache") pod "8aece7fc-7653-4788-b914-291f241caafa" (UID: "8aece7fc-7653-4788-b914-291f241caafa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:39:59.065772 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065675    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:39:59.066049 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.065995    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-home" (OuterVolumeSpecName: "home") pod "8aece7fc-7653-4788-b914-291f241caafa" (UID: "8aece7fc-7653-4788-b914-291f241caafa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:39:59.066248 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.066198    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aece7fc-7653-4788-b914-291f241caafa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8aece7fc-7653-4788-b914-291f241caafa" (UID: "8aece7fc-7653-4788-b914-291f241caafa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:39:59.067724 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.067690    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-dshm" (OuterVolumeSpecName: "dshm") pod "8aece7fc-7653-4788-b914-291f241caafa" (UID: "8aece7fc-7653-4788-b914-291f241caafa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:39:59.067810 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.067723    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aece7fc-7653-4788-b914-291f241caafa-kube-api-access-qg4m9" (OuterVolumeSpecName: "kube-api-access-qg4m9") pod "8aece7fc-7653-4788-b914-291f241caafa" (UID: "8aece7fc-7653-4788-b914-291f241caafa"). InnerVolumeSpecName "kube-api-access-qg4m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:39:59.068161 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.068135    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aece7fc-7653-4788-b914-291f241caafa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8aece7fc-7653-4788-b914-291f241caafa" (UID: "8aece7fc-7653-4788-b914-291f241caafa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:39:59.084190 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.084166    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8aece7fc-7653-4788-b914-291f241caafa" (UID: "8aece7fc-7653-4788-b914-291f241caafa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:39:59.094167 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.094142    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8aece7fc-7653-4788-b914-291f241caafa" (UID: "8aece7fc-7653-4788-b914-291f241caafa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:39:59.166396 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.166365    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:39:59.166396 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.166391    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aece7fc-7653-4788-b914-291f241caafa-cabundle-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:39:59.166396 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.166401    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aece7fc-7653-4788-b914-291f241caafa-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:39:59.166597 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.166410    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:39:59.166597 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.166418    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4m9\" (UniqueName: \"kubernetes.io/projected/8aece7fc-7653-4788-b914-291f241caafa-kube-api-access-qg4m9\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:39:59.166597 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.166427    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-kserve-provision-location\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:39:59.166597 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.166435    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aece7fc-7653-4788-b914-291f241caafa-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:39:59.918901 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.918875    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 16:39:59.919329 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.918872    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8aece7fc-7653-4788-b914-291f241caafa","Type":"ContainerDied","Data":"1319c31ce8b150ba27ac9feda3dc8f18a61aa9968c66206d6d0870e821bdb4ae"}
Sep 17 16:39:59.919329 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.919007    2555 scope.go:117] "RemoveContainer" containerID="712ba626c9d03ac566e5010594dd96b613110937141e9e34b28a9f3a1820e164"
Sep 17 16:39:59.929568 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.929548    2555 scope.go:117] "RemoveContainer" containerID="6ac5d3993a7f12ca501dd1a7999f1c61deb3aabc8fba70f7f49341a1782a99d2"
Sep 17 16:39:59.945948 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.945841    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 16:39:59.948276 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:39:59.948254    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 16:40:00.695076 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:40:00.695000    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aece7fc-7653-4788-b914-291f241caafa" path="/var/lib/kubelet/pods/8aece7fc-7653-4788-b914-291f241caafa/volumes"
Sep 17 16:40:03.323757 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:40:03.323718    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:40:13.322889 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:40:13.322844    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:40:23.323042 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:40:23.322999    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:40:33.323275 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:40:33.323233    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:40:43.323292 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:40:43.323242    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:40:53.323361 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:40:53.323310    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:41:03.323842 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:03.323794    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:41:13.333141 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:13.333103    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:41:13.341173 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:13.341142    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:41:17.866938 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:17.866903    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"]
Sep 17 16:41:32.888812 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:32.888700    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" containerID="cri-o://f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8" gracePeriod=45
Sep 17 16:41:33.333666 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.333628    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 16:41:33.632419 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.632396    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:41:33.705724 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.705695    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fca2adf-7f94-4e6f-8f33-e244985da649-cabundle-cert\") pod \"6fca2adf-7f94-4e6f-8f33-e244985da649\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") "
Sep 17 16:41:33.705866 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.705735    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rdxj\" (UniqueName: \"kubernetes.io/projected/6fca2adf-7f94-4e6f-8f33-e244985da649-kube-api-access-8rdxj\") pod \"6fca2adf-7f94-4e6f-8f33-e244985da649\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") "
Sep 17 16:41:33.705866 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.705756    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-dshm\") pod \"6fca2adf-7f94-4e6f-8f33-e244985da649\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") "
Sep 17 16:41:33.705866 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.705771    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-home\") pod \"6fca2adf-7f94-4e6f-8f33-e244985da649\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") "
Sep 17 16:41:33.705866 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.705845    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-model-cache\") pod \"6fca2adf-7f94-4e6f-8f33-e244985da649\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") "
Sep 17 16:41:33.705866 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.705860    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-kserve-provision-location\") pod \"6fca2adf-7f94-4e6f-8f33-e244985da649\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") "
Sep 17 16:41:33.706162 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.705900    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-tmp-dir\") pod \"6fca2adf-7f94-4e6f-8f33-e244985da649\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") "
Sep 17 16:41:33.706162 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.705938    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fca2adf-7f94-4e6f-8f33-e244985da649-tls-certs\") pod \"6fca2adf-7f94-4e6f-8f33-e244985da649\" (UID: \"6fca2adf-7f94-4e6f-8f33-e244985da649\") "
Sep 17 16:41:33.706162 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.706130    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fca2adf-7f94-4e6f-8f33-e244985da649" (UID: "6fca2adf-7f94-4e6f-8f33-e244985da649"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:33.706339 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.706136    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fca2adf-7f94-4e6f-8f33-e244985da649-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6fca2adf-7f94-4e6f-8f33-e244985da649" (UID: "6fca2adf-7f94-4e6f-8f33-e244985da649"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:41:33.706585 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.706559    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-home" (OuterVolumeSpecName: "home") pod "6fca2adf-7f94-4e6f-8f33-e244985da649" (UID: "6fca2adf-7f94-4e6f-8f33-e244985da649"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:33.708493 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.708455    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fca2adf-7f94-4e6f-8f33-e244985da649-kube-api-access-8rdxj" (OuterVolumeSpecName: "kube-api-access-8rdxj") pod "6fca2adf-7f94-4e6f-8f33-e244985da649" (UID: "6fca2adf-7f94-4e6f-8f33-e244985da649"). InnerVolumeSpecName "kube-api-access-8rdxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:41:33.708493 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.708481    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fca2adf-7f94-4e6f-8f33-e244985da649-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fca2adf-7f94-4e6f-8f33-e244985da649" (UID: "6fca2adf-7f94-4e6f-8f33-e244985da649"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:41:33.708682 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.708499    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-dshm" (OuterVolumeSpecName: "dshm") pod "6fca2adf-7f94-4e6f-8f33-e244985da649" (UID: "6fca2adf-7f94-4e6f-8f33-e244985da649"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:33.721102 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.721078    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fca2adf-7f94-4e6f-8f33-e244985da649" (UID: "6fca2adf-7f94-4e6f-8f33-e244985da649"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:33.727095 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.727070    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6fca2adf-7f94-4e6f-8f33-e244985da649" (UID: "6fca2adf-7f94-4e6f-8f33-e244985da649"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:33.807113 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.807088    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:41:33.807113 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.807108    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-kserve-provision-location\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:41:33.807252 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.807118    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:41:33.807252 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.807128    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fca2adf-7f94-4e6f-8f33-e244985da649-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:41:33.807252 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.807136    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fca2adf-7f94-4e6f-8f33-e244985da649-cabundle-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:41:33.807252 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.807144    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rdxj\" (UniqueName: \"kubernetes.io/projected/6fca2adf-7f94-4e6f-8f33-e244985da649-kube-api-access-8rdxj\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:41:33.807252 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.807152    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:41:33.807252 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:33.807160    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fca2adf-7f94-4e6f-8f33-e244985da649-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:41:34.398328 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.398294    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerID="f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8" exitCode=0
Sep 17 16:41:34.398909 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.398342    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"
Sep 17 16:41:34.398909 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.398352    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" event={"ID":"6fca2adf-7f94-4e6f-8f33-e244985da649","Type":"ContainerDied","Data":"f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8"}
Sep 17 16:41:34.398909 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.398406    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb" event={"ID":"6fca2adf-7f94-4e6f-8f33-e244985da649","Type":"ContainerDied","Data":"522929fcc43b4038ccee71352268cea9ad60ad3abafbf3f60c0c83f575a8a204"}
Sep 17 16:41:34.398909 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.398439    2555 scope.go:117] "RemoveContainer" containerID="f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8"
Sep 17 16:41:34.407668 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.407652    2555 scope.go:117] "RemoveContainer" containerID="7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f"
Sep 17 16:41:34.415388 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.415370    2555 scope.go:117] "RemoveContainer" containerID="f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8"
Sep 17 16:41:34.415631 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:41:34.415613    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8\": container with ID starting with f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8 not found: ID does not exist" containerID="f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8"
Sep 17 16:41:34.415676 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.415639    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8"} err="failed to get container status \"f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8\": rpc error: code = NotFound desc = could not find container \"f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8\": container with ID starting with f0446dfa7b0aec7fd3ac9c48db69c938623ea49d413ce7e8db4d3984201111f8 not found: ID does not exist"
Sep 17 16:41:34.415676 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.415653    2555 scope.go:117] "RemoveContainer" containerID="7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f"
Sep 17 16:41:34.415894 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:41:34.415879    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f\": container with ID starting with 7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f not found: ID does not exist" containerID="7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f"
Sep 17 16:41:34.415938 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.415900    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f"} err="failed to get container status \"7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f\": rpc error: code = NotFound desc = could not find container \"7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f\": container with ID starting with 7e9308ce8fcb0342f69dd3e3dc2cee74eb8254034e9b6ea5c3af5a759480bc0f not found: ID does not exist"
Sep 17 16:41:34.422555 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.422532    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"]
Sep 17 16:41:34.426111 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.426091    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-f4744dcf4-7n2fb"]
Sep 17 16:41:34.693620 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:41:34.693545    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" path="/var/lib/kubelet/pods/6fca2adf-7f94-4e6f-8f33-e244985da649/volumes"
Sep 17 16:42:46.673976 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.673941    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"]
Sep 17 16:42:46.674587 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.674346    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="storage-initializer"
Sep 17 16:42:46.674587 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.674364    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="storage-initializer"
Sep 17 16:42:46.674587 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.674374    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main"
Sep 17 16:42:46.674587 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.674381    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main"
Sep 17 16:42:46.674587 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.674407    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="storage-initializer"
Sep 17 16:42:46.674587 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.674416    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="storage-initializer"
Sep 17 16:42:46.674587 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.674425    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main"
Sep 17 16:42:46.674587 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.674432    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main"
Sep 17 16:42:46.674587 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.674511    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fca2adf-7f94-4e6f-8f33-e244985da649" containerName="main"
Sep 17 16:42:46.674587 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.674523    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aece7fc-7653-4788-b914-291f241caafa" containerName="main"
Sep 17 16:42:46.676530 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.676509    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.679307 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.679271    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 16:42:46.679425 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.679318    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 16:42:46.679425 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.679374    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-tpv2n\""
Sep 17 16:42:46.680500 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.680463    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 16:42:46.686423 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.686402    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"]
Sep 17 16:42:46.714255 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.714222    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcj4r\" (UniqueName: \"kubernetes.io/projected/43585162-931f-4878-b597-641b9401bb57-kube-api-access-mcj4r\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.714361 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.714266    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.714361 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.714302    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.714361 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.714336    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.714487 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.714373    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.714487 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.714404    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43585162-931f-4878-b597-641b9401bb57-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.814847 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.814801    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.814981 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.814857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.814981 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.814899    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.814981 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.814931    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43585162-931f-4878-b597-641b9401bb57-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.815137 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.815007    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcj4r\" (UniqueName: \"kubernetes.io/projected/43585162-931f-4878-b597-641b9401bb57-kube-api-access-mcj4r\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.815137 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.815031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.815263 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.815166    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.815347 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.815328    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.815406 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.815330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.817085 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.817065    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.817390 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.817375    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43585162-931f-4878-b597-641b9401bb57-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.823571 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.823552    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcj4r\" (UniqueName: \"kubernetes.io/projected/43585162-931f-4878-b597-641b9401bb57-kube-api-access-mcj4r\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:46.989331 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:46.989243    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:47.262411 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:47.262384    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"]
Sep 17 16:42:47.264367 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:42:47.264339    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43585162_931f_4878_b597_641b9401bb57.slice/crio-8216bae86f7ff750220781721a384b804bc8e7ba444b83051906f1f6cf728fd5 WatchSource:0}: Error finding container 8216bae86f7ff750220781721a384b804bc8e7ba444b83051906f1f6cf728fd5: Status 404 returned error can't find the container with id 8216bae86f7ff750220781721a384b804bc8e7ba444b83051906f1f6cf728fd5
Sep 17 16:42:47.755815 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:47.755762    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6" event={"ID":"43585162-931f-4878-b597-641b9401bb57","Type":"ContainerStarted","Data":"ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd"}
Sep 17 16:42:47.756328 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:47.755862    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6" event={"ID":"43585162-931f-4878-b597-641b9401bb57","Type":"ContainerStarted","Data":"8216bae86f7ff750220781721a384b804bc8e7ba444b83051906f1f6cf728fd5"}
Sep 17 16:42:47.776464 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:47.776410    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6" podStartSLOduration=1.7763952170000001 podStartE2EDuration="1.776395217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:47.774698259 +0000 UTC m=+3849.665558824" watchObservedRunningTime="2026-09-17 16:42:47.776395217 +0000 UTC m=+3849.667255761"
Sep 17 16:42:47.990054 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:47.990021    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:56.989674 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:56.989634    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:57.002213 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:57.002189    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:42:57.811133 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:42:57.811100    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:43:20.424773 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.424733    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"]
Sep 17 16:43:20.433849 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:43:20.433784    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:43:20.433849 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:43:20.433849 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:43:20.433849 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd" cmd=["/bin/sleep","15"]
Sep 17 16:43:20.434192 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.433949    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6" message="" err=<
Sep 17 16:43:20.434192 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:43:20.434192 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:43:20.434192 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 16:43:20.434192 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:43:20.433975    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:43:20.434192 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:43:20.434192 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:43:20.434192 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6" podUID="43585162-931f-4878-b597-641b9401bb57" containerName="main" containerID="cri-o://ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd"
Sep 17 16:43:20.434192 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.434030    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6" podUID="43585162-931f-4878-b597-641b9401bb57" containerName="main" containerID="cri-o://ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd" gracePeriod=60
Sep 17 16:43:20.767220 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.767196    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:43:20.860675 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.860648    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43585162-931f-4878-b597-641b9401bb57-tls-certs\") pod \"43585162-931f-4878-b597-641b9401bb57\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") "
Sep 17 16:43:20.860798 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.860687    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-home\") pod \"43585162-931f-4878-b597-641b9401bb57\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") "
Sep 17 16:43:20.860798 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.860743    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-dshm\") pod \"43585162-931f-4878-b597-641b9401bb57\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") "
Sep 17 16:43:20.860798 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.860769    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcj4r\" (UniqueName: \"kubernetes.io/projected/43585162-931f-4878-b597-641b9401bb57-kube-api-access-mcj4r\") pod \"43585162-931f-4878-b597-641b9401bb57\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") "
Sep 17 16:43:20.860798 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.860792    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-model-cache\") pod \"43585162-931f-4878-b597-641b9401bb57\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") "
Sep 17 16:43:20.861036 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.860815    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-tmp-dir\") pod \"43585162-931f-4878-b597-641b9401bb57\" (UID: \"43585162-931f-4878-b597-641b9401bb57\") "
Sep 17 16:43:20.861036 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.861012    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-home" (OuterVolumeSpecName: "home") pod "43585162-931f-4878-b597-641b9401bb57" (UID: "43585162-931f-4878-b597-641b9401bb57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:43:20.861147 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.861051    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-model-cache" (OuterVolumeSpecName: "model-cache") pod "43585162-931f-4878-b597-641b9401bb57" (UID: "43585162-931f-4878-b597-641b9401bb57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:43:20.861147 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.861131    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43585162-931f-4878-b597-641b9401bb57" (UID: "43585162-931f-4878-b597-641b9401bb57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:43:20.863017 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.862993    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43585162-931f-4878-b597-641b9401bb57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43585162-931f-4878-b597-641b9401bb57" (UID: "43585162-931f-4878-b597-641b9401bb57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:20.863017 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.862999    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43585162-931f-4878-b597-641b9401bb57-kube-api-access-mcj4r" (OuterVolumeSpecName: "kube-api-access-mcj4r") pod "43585162-931f-4878-b597-641b9401bb57" (UID: "43585162-931f-4878-b597-641b9401bb57"). InnerVolumeSpecName "kube-api-access-mcj4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:20.863159 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.863085    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-dshm" (OuterVolumeSpecName: "dshm") pod "43585162-931f-4878-b597-641b9401bb57" (UID: "43585162-931f-4878-b597-641b9401bb57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:43:20.909750 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.909721    2555 generic.go:356] "Generic (PLEG): container finished" podID="43585162-931f-4878-b597-641b9401bb57" containerID="ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd" exitCode=0
Sep 17 16:43:20.909876 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.909763    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"
Sep 17 16:43:20.909876 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.909809    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6" event={"ID":"43585162-931f-4878-b597-641b9401bb57","Type":"ContainerDied","Data":"ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd"}
Sep 17 16:43:20.909969 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.909880    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6" event={"ID":"43585162-931f-4878-b597-641b9401bb57","Type":"ContainerDied","Data":"8216bae86f7ff750220781721a384b804bc8e7ba444b83051906f1f6cf728fd5"}
Sep 17 16:43:20.909969 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.909903    2555 scope.go:117] "RemoveContainer" containerID="ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd"
Sep 17 16:43:20.918178 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.918162    2555 scope.go:117] "RemoveContainer" containerID="ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd"
Sep 17 16:43:20.918399 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:43:20.918381    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd\": container with ID starting with ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd not found: ID does not exist" containerID="ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd"
Sep 17 16:43:20.918447 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.918407    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd"} err="failed to get container status \"ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd\": rpc error: code = NotFound desc = could not find container \"ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd\": container with ID starting with ed856b1bca1d8d8fd9aa58e3392123ab5a28aebe3c7d9f4c86477340647015bd not found: ID does not exist"
Sep 17 16:43:20.935997 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.935971    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"]
Sep 17 16:43:20.942500 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.942480    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8f2x6"]
Sep 17 16:43:20.961849 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.961797    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43585162-931f-4878-b597-641b9401bb57-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:43:20.961849 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.961815    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:43:20.961849 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.961837    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:43:20.961849 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.961849    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcj4r\" (UniqueName: \"kubernetes.io/projected/43585162-931f-4878-b597-641b9401bb57-kube-api-access-mcj4r\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:43:20.962011 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.961858    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:43:20.962011 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:20.961867    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43585162-931f-4878-b597-641b9401bb57-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:43:22.694806 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:22.694769    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43585162-931f-4878-b597-641b9401bb57" path="/var/lib/kubelet/pods/43585162-931f-4878-b597-641b9401bb57/volumes"
Sep 17 16:43:39.026140 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:39.026109    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:43:39.035739 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:43:39.035719    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h7dwx_ab9584e9-4c5a-4603-af6b-e26845d8bf04/ovn-acl-logging/0.log"
Sep 17 16:44:19.019231 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.019189    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"]
Sep 17 16:44:19.019780 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.019574    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43585162-931f-4878-b597-641b9401bb57" containerName="main"
Sep 17 16:44:19.019780 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.019591    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="43585162-931f-4878-b597-641b9401bb57" containerName="main"
Sep 17 16:44:19.019780 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.019692    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="43585162-931f-4878-b597-641b9401bb57" containerName="main"
Sep 17 16:44:19.022782 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.022761    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.025654 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.025635    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 16:44:19.025973 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.025954    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 16:44:19.026042 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.025997    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-ktq5d\""
Sep 17 16:44:19.026209 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.026196    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 16:44:19.026262 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.026239    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 16:44:19.026662 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.026647    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:44:19.036489 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.036469    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"]
Sep 17 16:44:19.177078 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.177045    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-kserve-provision-location\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.177273 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.177086    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373f601b-15e8-4771-84c1-6b9ec34fda82-tls-certs\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.177273 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.177106    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/373f601b-15e8-4771-84c1-6b9ec34fda82-cabundle-cert\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.177273 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.177125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnkwz\" (UniqueName: \"kubernetes.io/projected/373f601b-15e8-4771-84c1-6b9ec34fda82-kube-api-access-jnkwz\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.177273 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.177204    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-model-cache\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.177273 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.177262    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-tmp-dir\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.177532 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.177301    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-home\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.177532 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.177332    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-dshm\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.278626 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.278546    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-kserve-provision-location\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.278626 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.278581    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373f601b-15e8-4771-84c1-6b9ec34fda82-tls-certs\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.278626 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.278601    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/373f601b-15e8-4771-84c1-6b9ec34fda82-cabundle-cert\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.278626 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.278621    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnkwz\" (UniqueName: \"kubernetes.io/projected/373f601b-15e8-4771-84c1-6b9ec34fda82-kube-api-access-jnkwz\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.278996 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.278648    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-model-cache\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.278996 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.278775    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-tmp-dir\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.278996 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.278866    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-home\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.278996 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.278903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-dshm\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.278996 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.278993    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-kserve-provision-location\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.279233 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.279070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-model-cache\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.279282 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.279242    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-home\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.279282 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.279252    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-tmp-dir\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.279424 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.279407    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/373f601b-15e8-4771-84c1-6b9ec34fda82-cabundle-cert\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.281036 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.281020    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-dshm\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.281305 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.281288    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373f601b-15e8-4771-84c1-6b9ec34fda82-tls-certs\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.288293 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.288269    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnkwz\" (UniqueName: \"kubernetes.io/projected/373f601b-15e8-4771-84c1-6b9ec34fda82-kube-api-access-jnkwz\") pod \"prestop-hook-test-kserve-6989b4f56b-45phl\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.332969 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.332928    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:19.611792 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:19.611761    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"]
Sep 17 16:44:19.615346 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:44:19.613006    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373f601b_15e8_4771_84c1_6b9ec34fda82.slice/crio-39912b167c8ad7639b1abf3b7b7e84d75630d0a836f0cf3fd01da0198d6ac9db WatchSource:0}: Error finding container 39912b167c8ad7639b1abf3b7b7e84d75630d0a836f0cf3fd01da0198d6ac9db: Status 404 returned error can't find the container with id 39912b167c8ad7639b1abf3b7b7e84d75630d0a836f0cf3fd01da0198d6ac9db
Sep 17 16:44:20.194465 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:20.194421    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl" event={"ID":"373f601b-15e8-4771-84c1-6b9ec34fda82","Type":"ContainerStarted","Data":"b1c4ce794c6e0065f893e5981bfbada42e541615b6b79453b6970987a7a7cb3e"}
Sep 17 16:44:20.194877 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:20.194477    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl" event={"ID":"373f601b-15e8-4771-84c1-6b9ec34fda82","Type":"ContainerStarted","Data":"39912b167c8ad7639b1abf3b7b7e84d75630d0a836f0cf3fd01da0198d6ac9db"}
Sep 17 16:44:22.205096 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:22.205064    2555 generic.go:356] "Generic (PLEG): container finished" podID="373f601b-15e8-4771-84c1-6b9ec34fda82" containerID="b1c4ce794c6e0065f893e5981bfbada42e541615b6b79453b6970987a7a7cb3e" exitCode=0
Sep 17 16:44:22.205524 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:22.205133    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl" event={"ID":"373f601b-15e8-4771-84c1-6b9ec34fda82","Type":"ContainerDied","Data":"b1c4ce794c6e0065f893e5981bfbada42e541615b6b79453b6970987a7a7cb3e"}
Sep 17 16:44:23.213224 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:23.213189    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl" event={"ID":"373f601b-15e8-4771-84c1-6b9ec34fda82","Type":"ContainerStarted","Data":"b65f8ebc5739d6625906f6ccfaa6014471f72aa32e9de6d187d976ddc7d8dcd8"}
Sep 17 16:44:23.239049 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:23.238996    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl" podStartSLOduration=3.090035667 podStartE2EDuration="5.238976511s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.148940844s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:23.237003182 +0000 UTC m=+3945.127863752" watchObservedRunningTime="2026-09-17 16:44:23.238976511 +0000 UTC m=+3945.129837059"
Sep 17 16:44:23.333983 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:23.333955    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:25.985477 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:25.985440    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"]
Sep 17 16:44:25.989157 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:25.989123    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:25.993132 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:25.993108    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 16:44:25.993272 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:25.993107    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 16:44:25.993272 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:25.993108    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 16:44:25.993272 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:25.993244    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-hd5j4\""
Sep 17 16:44:25.999278 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:25.999210    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"]
Sep 17 16:44:26.030927 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.030892    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.031069 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.030944    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.031069 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.031018    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.031146 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.031071    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.031146 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.031124    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.031221 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.031151    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfchc\" (UniqueName: \"kubernetes.io/projected/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-kube-api-access-dfchc\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.132166 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.132130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.132336 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.132183    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.132384 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.132323    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.132384 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.132368    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfchc\" (UniqueName: \"kubernetes.io/projected/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-kube-api-access-dfchc\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.132463 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.132437    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.132506 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.132471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.132635 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.132601    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.132744 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.132657    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.132870 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.132843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.134879 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.134860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.134971 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.134952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.142962 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.142939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfchc\" (UniqueName: \"kubernetes.io/projected/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-kube-api-access-dfchc\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-2647g\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.301868 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.301759    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:26.577205 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:26.577178    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"]
Sep 17 16:44:26.578750 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:44:26.578721    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adaa020_c5d6_40a2_9f1e_b9cc88e62f87.slice/crio-9cac5a7c3b5236f3a40838ed0867ca2cf8a56d2a7e7e40b3e96e4053ba3780eb WatchSource:0}: Error finding container 9cac5a7c3b5236f3a40838ed0867ca2cf8a56d2a7e7e40b3e96e4053ba3780eb: Status 404 returned error can't find the container with id 9cac5a7c3b5236f3a40838ed0867ca2cf8a56d2a7e7e40b3e96e4053ba3780eb
Sep 17 16:44:27.235743 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:27.235700    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g" event={"ID":"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87","Type":"ContainerStarted","Data":"f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42"}
Sep 17 16:44:27.236170 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:27.235765    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g" event={"ID":"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87","Type":"ContainerStarted","Data":"9cac5a7c3b5236f3a40838ed0867ca2cf8a56d2a7e7e40b3e96e4053ba3780eb"}
Sep 17 16:44:27.301943 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:27.301898    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:29.333411 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:29.333365    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:44:29.334597 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:29.334570    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl" podUID="373f601b-15e8-4771-84c1-6b9ec34fda82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 16:44:36.302049 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:36.301966    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:36.314661 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:36.314636    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:36.340320 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:36.340277    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g" podStartSLOduration=11.340263338 podStartE2EDuration="11.340263338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:27.283928487 +0000 UTC m=+3949.174789031" watchObservedRunningTime="2026-09-17 16:44:36.340263338 +0000 UTC m=+3958.231123881"
Sep 17 16:44:37.302321 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:37.302287    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:44:39.333405 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:39.333362    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl" podUID="373f601b-15e8-4771-84c1-6b9ec34fda82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 16:44:41.206178 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:41.206136    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"]
Sep 17 16:44:56.233309 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:44:56.233268    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl" podUID="373f601b-15e8-4771-84c1-6b9ec34fda82" containerName="main" containerID="cri-o://b65f8ebc5739d6625906f6ccfaa6014471f72aa32e9de6d187d976ddc7d8dcd8" gracePeriod=45
Sep 17 16:45:00.320627 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:00.320591    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"]
Sep 17 16:45:00.329543 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:45:00.329490    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:45:00.329543 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:45:00.329543 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:45:00.329543 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42" cmd=["/bin/sleep","15"]
Sep 17 16:45:00.329875 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:00.329564    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g" message="" err=<
Sep 17 16:45:00.329875 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:45:00.329875 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:45:00.329875 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 16:45:00.329875 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:45:00.329581    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:45:00.329875 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:45:00.329875 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:45:00.329875 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g" podUID="9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" containerName="main" containerID="cri-o://f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42"
Sep 17 16:45:00.329875 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:00.329625    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g" podUID="9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" containerName="main" containerID="cri-o://f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42" gracePeriod=60
Sep 17 16:45:01.081519 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.081484    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:45:01.215630 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.215551    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tmp-dir\") pod \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") "
Sep 17 16:45:01.215630 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.215598    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tls-certs\") pod \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") "
Sep 17 16:45:01.215877 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.215636    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-model-cache\") pod \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") "
Sep 17 16:45:01.215877 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.215659    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-home\") pod \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") "
Sep 17 16:45:01.215877 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.215682    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-dshm\") pod \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") "
Sep 17 16:45:01.215877 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.215705    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfchc\" (UniqueName: \"kubernetes.io/projected/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-kube-api-access-dfchc\") pod \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\" (UID: \"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87\") "
Sep 17 16:45:01.216103 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.215912    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" (UID: "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:45:01.216103 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.215959    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-model-cache" (OuterVolumeSpecName: "model-cache") pod "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" (UID: "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:45:01.216103 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.215986    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-home" (OuterVolumeSpecName: "home") pod "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" (UID: "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:45:01.216103 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.216070    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:01.216103 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.216090    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:01.218078 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.218046    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" (UID: "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:45:01.218078 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.218067    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-kube-api-access-dfchc" (OuterVolumeSpecName: "kube-api-access-dfchc") pod "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" (UID: "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87"). InnerVolumeSpecName "kube-api-access-dfchc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:45:01.218256 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.218180    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-dshm" (OuterVolumeSpecName: "dshm") pod "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" (UID: "9adaa020-c5d6-40a2-9f1e-b9cc88e62f87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:45:01.317399 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.317375    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:01.317399 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.317399    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:01.317580 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.317411    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:01.317580 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.317423    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfchc\" (UniqueName: \"kubernetes.io/projected/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87-kube-api-access-dfchc\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:01.414607 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.414573    2555 generic.go:356] "Generic (PLEG): container finished" podID="9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" containerID="f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42" exitCode=0
Sep 17 16:45:01.415013 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.414611    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"
Sep 17 16:45:01.415013 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.414657    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g" event={"ID":"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87","Type":"ContainerDied","Data":"f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42"}
Sep 17 16:45:01.415013 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.414717    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g" event={"ID":"9adaa020-c5d6-40a2-9f1e-b9cc88e62f87","Type":"ContainerDied","Data":"9cac5a7c3b5236f3a40838ed0867ca2cf8a56d2a7e7e40b3e96e4053ba3780eb"}
Sep 17 16:45:01.415013 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.414755    2555 scope.go:117] "RemoveContainer" containerID="f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42"
Sep 17 16:45:01.424258 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.424240    2555 scope.go:117] "RemoveContainer" containerID="f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42"
Sep 17 16:45:01.424565 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:45:01.424548    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42\": container with ID starting with f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42 not found: ID does not exist" containerID="f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42"
Sep 17 16:45:01.424624 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.424575    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42"} err="failed to get container status \"f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42\": rpc error: code = NotFound desc = could not find container \"f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42\": container with ID starting with f10546f7c7ad23d994397a9540ca5bea129fe399216e2552788965ec736ecb42 not found: ID does not exist"
Sep 17 16:45:01.448103 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.448078    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"]
Sep 17 16:45:01.450222 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:01.450202    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-2647g"]
Sep 17 16:45:02.694840 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:02.694801    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" path="/var/lib/kubelet/pods/9adaa020-c5d6-40a2-9f1e-b9cc88e62f87/volumes"
Sep 17 16:45:41.613755 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.613729    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6989b4f56b-45phl_373f601b-15e8-4771-84c1-6b9ec34fda82/main/0.log"
Sep 17 16:45:41.614190 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.614093    2555 generic.go:356] "Generic (PLEG): container finished" podID="373f601b-15e8-4771-84c1-6b9ec34fda82" containerID="b65f8ebc5739d6625906f6ccfaa6014471f72aa32e9de6d187d976ddc7d8dcd8" exitCode=137
Sep 17 16:45:41.614190 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.614132    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl" event={"ID":"373f601b-15e8-4771-84c1-6b9ec34fda82","Type":"ContainerDied","Data":"b65f8ebc5739d6625906f6ccfaa6014471f72aa32e9de6d187d976ddc7d8dcd8"}
Sep 17 16:45:41.875419 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.875396    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6989b4f56b-45phl_373f601b-15e8-4771-84c1-6b9ec34fda82/main/0.log"
Sep 17 16:45:41.878050 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.878031    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:45:41.935025 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.934997    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-tmp-dir\") pod \"373f601b-15e8-4771-84c1-6b9ec34fda82\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") "
Sep 17 16:45:41.935157 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935035    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/373f601b-15e8-4771-84c1-6b9ec34fda82-cabundle-cert\") pod \"373f601b-15e8-4771-84c1-6b9ec34fda82\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") "
Sep 17 16:45:41.935157 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935060    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-kserve-provision-location\") pod \"373f601b-15e8-4771-84c1-6b9ec34fda82\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") "
Sep 17 16:45:41.935157 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935093    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnkwz\" (UniqueName: \"kubernetes.io/projected/373f601b-15e8-4771-84c1-6b9ec34fda82-kube-api-access-jnkwz\") pod \"373f601b-15e8-4771-84c1-6b9ec34fda82\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") "
Sep 17 16:45:41.935157 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935144    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373f601b-15e8-4771-84c1-6b9ec34fda82-tls-certs\") pod \"373f601b-15e8-4771-84c1-6b9ec34fda82\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") "
Sep 17 16:45:41.935392 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935164    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-model-cache\") pod \"373f601b-15e8-4771-84c1-6b9ec34fda82\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") "
Sep 17 16:45:41.935392 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935186    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-home\") pod \"373f601b-15e8-4771-84c1-6b9ec34fda82\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") "
Sep 17 16:45:41.935392 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935211    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-dshm\") pod \"373f601b-15e8-4771-84c1-6b9ec34fda82\" (UID: \"373f601b-15e8-4771-84c1-6b9ec34fda82\") "
Sep 17 16:45:41.935617 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935390    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373f601b-15e8-4771-84c1-6b9ec34fda82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "373f601b-15e8-4771-84c1-6b9ec34fda82" (UID: "373f601b-15e8-4771-84c1-6b9ec34fda82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:41.935617 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935511    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-model-cache" (OuterVolumeSpecName: "model-cache") pod "373f601b-15e8-4771-84c1-6b9ec34fda82" (UID: "373f601b-15e8-4771-84c1-6b9ec34fda82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:45:41.935734 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935686    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-home" (OuterVolumeSpecName: "home") pod "373f601b-15e8-4771-84c1-6b9ec34fda82" (UID: "373f601b-15e8-4771-84c1-6b9ec34fda82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:45:41.935922 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935902    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:41.936005 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935927    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:41.936005 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.935941    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/373f601b-15e8-4771-84c1-6b9ec34fda82-cabundle-cert\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:41.936116 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.936009    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "373f601b-15e8-4771-84c1-6b9ec34fda82" (UID: "373f601b-15e8-4771-84c1-6b9ec34fda82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:45:41.937855 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.937792    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373f601b-15e8-4771-84c1-6b9ec34fda82-kube-api-access-jnkwz" (OuterVolumeSpecName: "kube-api-access-jnkwz") pod "373f601b-15e8-4771-84c1-6b9ec34fda82" (UID: "373f601b-15e8-4771-84c1-6b9ec34fda82"). InnerVolumeSpecName "kube-api-access-jnkwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:45:41.937855 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.937794    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373f601b-15e8-4771-84c1-6b9ec34fda82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "373f601b-15e8-4771-84c1-6b9ec34fda82" (UID: "373f601b-15e8-4771-84c1-6b9ec34fda82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:45:41.937987 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.937944    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-dshm" (OuterVolumeSpecName: "dshm") pod "373f601b-15e8-4771-84c1-6b9ec34fda82" (UID: "373f601b-15e8-4771-84c1-6b9ec34fda82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:45:41.957558 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:41.957532    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "373f601b-15e8-4771-84c1-6b9ec34fda82" (UID: "373f601b-15e8-4771-84c1-6b9ec34fda82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:45:42.037041 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.037022    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373f601b-15e8-4771-84c1-6b9ec34fda82-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.037041 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.037041    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.037167 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.037066    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.037167 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.037074    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373f601b-15e8-4771-84c1-6b9ec34fda82-kserve-provision-location\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.037167 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.037083    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnkwz\" (UniqueName: \"kubernetes.io/projected/373f601b-15e8-4771-84c1-6b9ec34fda82-kube-api-access-jnkwz\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.619750 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.619722    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6989b4f56b-45phl_373f601b-15e8-4771-84c1-6b9ec34fda82/main/0.log"
Sep 17 16:45:42.620190 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.620169    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"
Sep 17 16:45:42.620259 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.620161    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl" event={"ID":"373f601b-15e8-4771-84c1-6b9ec34fda82","Type":"ContainerDied","Data":"39912b167c8ad7639b1abf3b7b7e84d75630d0a836f0cf3fd01da0198d6ac9db"}
Sep 17 16:45:42.620300 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.620287    2555 scope.go:117] "RemoveContainer" containerID="b65f8ebc5739d6625906f6ccfaa6014471f72aa32e9de6d187d976ddc7d8dcd8"
Sep 17 16:45:42.629768 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.629753    2555 scope.go:117] "RemoveContainer" containerID="b1c4ce794c6e0065f893e5981bfbada42e541615b6b79453b6970987a7a7cb3e"
Sep 17 16:45:42.646573 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.646553    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"]
Sep 17 16:45:42.650481 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.650457    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-45phl"]
Sep 17 16:45:42.694407 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:45:42.694375    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373f601b-15e8-4771-84c1-6b9ec34fda82" path="/var/lib/kubelet/pods/373f601b-15e8-4771-84c1-6b9ec34fda82/volumes"
Sep 17 16:46:27.793419 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.793382    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"]
Sep 17 16:46:27.793914 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.793665    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373f601b-15e8-4771-84c1-6b9ec34fda82" containerName="storage-initializer"
Sep 17 16:46:27.793914 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.793678    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="373f601b-15e8-4771-84c1-6b9ec34fda82" containerName="storage-initializer"
Sep 17 16:46:27.793914 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.793701    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373f601b-15e8-4771-84c1-6b9ec34fda82" containerName="main"
Sep 17 16:46:27.793914 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.793707    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="373f601b-15e8-4771-84c1-6b9ec34fda82" containerName="main"
Sep 17 16:46:27.793914 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.793714    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" containerName="main"
Sep 17 16:46:27.793914 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.793720    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" containerName="main"
Sep 17 16:46:27.793914 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.793769    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="373f601b-15e8-4771-84c1-6b9ec34fda82" containerName="main"
Sep 17 16:46:27.793914 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.793780    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adaa020-c5d6-40a2-9f1e-b9cc88e62f87" containerName="main"
Sep 17 16:46:27.796530 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.796513    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.800764 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.800733    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 16:46:27.800935 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.800802    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 16:46:27.800935 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.800813    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-6qjxj\""
Sep 17 16:46:27.800935 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.800849    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 16:46:27.806068 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.806046    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"]
Sep 17 16:46:27.870102 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.870066    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.870102 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.870101    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.870312 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.870131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f715c7-a10b-4da9-9b21-08eb1070e746-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.870312 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.870191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.870312 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.870264    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hvv4\" (UniqueName: \"kubernetes.io/projected/49f715c7-a10b-4da9-9b21-08eb1070e746-kube-api-access-7hvv4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.870312 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.870297    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.971175 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.971152    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hvv4\" (UniqueName: \"kubernetes.io/projected/49f715c7-a10b-4da9-9b21-08eb1070e746-kube-api-access-7hvv4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.971316 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.971183    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.971316 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.971205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.971316 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.971220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.971316 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.971238    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f715c7-a10b-4da9-9b21-08eb1070e746-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.971316 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.971268    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.971578 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.971554    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.971627 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.971603    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.971667 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.971645    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.973444 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.973422    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.973896 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.973877    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f715c7-a10b-4da9-9b21-08eb1070e746-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:27.988026 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:27.988005    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hvv4\" (UniqueName: \"kubernetes.io/projected/49f715c7-a10b-4da9-9b21-08eb1070e746-kube-api-access-7hvv4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v5jl6\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:28.107190 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:28.107161    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:28.379651 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:28.379619    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"]
Sep 17 16:46:28.379985 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:46:28.379963    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f715c7_a10b_4da9_9b21_08eb1070e746.slice/crio-406cf2a76c67c3c0aaa2873f2093a7234651d4f513e413c2e309e5511461951e WatchSource:0}: Error finding container 406cf2a76c67c3c0aaa2873f2093a7234651d4f513e413c2e309e5511461951e: Status 404 returned error can't find the container with id 406cf2a76c67c3c0aaa2873f2093a7234651d4f513e413c2e309e5511461951e
Sep 17 16:46:28.835545 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:28.835514    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6" event={"ID":"49f715c7-a10b-4da9-9b21-08eb1070e746","Type":"ContainerStarted","Data":"abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1"}
Sep 17 16:46:28.835940 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:28.835561    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6" event={"ID":"49f715c7-a10b-4da9-9b21-08eb1070e746","Type":"ContainerStarted","Data":"406cf2a76c67c3c0aaa2873f2093a7234651d4f513e413c2e309e5511461951e"}
Sep 17 16:46:28.857808 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:28.857766    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6" podStartSLOduration=1.857752526 podStartE2EDuration="1.857752526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:46:28.856431287 +0000 UTC m=+4070.747291831" watchObservedRunningTime="2026-09-17 16:46:28.857752526 +0000 UTC m=+4070.748613069"
Sep 17 16:46:29.107914 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:29.107813    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:38.108242 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:38.108207    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:38.120779 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:38.120751    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:38.893251 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:38.893215    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:42.799096 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:42.799065    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"]
Sep 17 16:46:42.807704 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:46:42.807664    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:46:42.807704 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:46:42.807704 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:46:42.807704 ip-10-0-135-243 kubenswrapper[2555]:  > containerID="abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1" cmd=["/bin/sleep","15"]
Sep 17 16:46:42.808009 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:42.807726    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6" message="" err=<
Sep 17 16:46:42.808009 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:46:42.808009 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:46:42.808009 ip-10-0-135-243 kubenswrapper[2555]:  >
Sep 17 16:46:42.808009 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:46:42.807741    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:46:42.808009 ip-10-0-135-243 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:46:42.808009 ip-10-0-135-243 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 16:46:42.808009 ip-10-0-135-243 kubenswrapper[2555]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6" podUID="49f715c7-a10b-4da9-9b21-08eb1070e746" containerName="main" containerID="cri-o://abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1"
Sep 17 16:46:42.808009 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:42.807784    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6" podUID="49f715c7-a10b-4da9-9b21-08eb1070e746" containerName="main" containerID="cri-o://abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1" gracePeriod=60
Sep 17 16:46:42.883324 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:42.883286    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6" podUID="49f715c7-a10b-4da9-9b21-08eb1070e746" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 16:46:43.549314 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.549289    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:43.686235 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.686158    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-dshm\") pod \"49f715c7-a10b-4da9-9b21-08eb1070e746\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") "
Sep 17 16:46:43.686235 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.686212    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-tmp-dir\") pod \"49f715c7-a10b-4da9-9b21-08eb1070e746\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") "
Sep 17 16:46:43.686235 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.686232    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hvv4\" (UniqueName: \"kubernetes.io/projected/49f715c7-a10b-4da9-9b21-08eb1070e746-kube-api-access-7hvv4\") pod \"49f715c7-a10b-4da9-9b21-08eb1070e746\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") "
Sep 17 16:46:43.686437 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.686254    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-model-cache\") pod \"49f715c7-a10b-4da9-9b21-08eb1070e746\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") "
Sep 17 16:46:43.686437 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.686290    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-home\") pod \"49f715c7-a10b-4da9-9b21-08eb1070e746\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") "
Sep 17 16:46:43.686437 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.686306    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f715c7-a10b-4da9-9b21-08eb1070e746-tls-certs\") pod \"49f715c7-a10b-4da9-9b21-08eb1070e746\" (UID: \"49f715c7-a10b-4da9-9b21-08eb1070e746\") "
Sep 17 16:46:43.686600 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.686452    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49f715c7-a10b-4da9-9b21-08eb1070e746" (UID: "49f715c7-a10b-4da9-9b21-08eb1070e746"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:43.686600 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.686484    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-model-cache" (OuterVolumeSpecName: "model-cache") pod "49f715c7-a10b-4da9-9b21-08eb1070e746" (UID: "49f715c7-a10b-4da9-9b21-08eb1070e746"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:43.686600 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.686575    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-home" (OuterVolumeSpecName: "home") pod "49f715c7-a10b-4da9-9b21-08eb1070e746" (UID: "49f715c7-a10b-4da9-9b21-08eb1070e746"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:43.688479 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.688457    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-dshm" (OuterVolumeSpecName: "dshm") pod "49f715c7-a10b-4da9-9b21-08eb1070e746" (UID: "49f715c7-a10b-4da9-9b21-08eb1070e746"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:43.688479 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.688470    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f715c7-a10b-4da9-9b21-08eb1070e746-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49f715c7-a10b-4da9-9b21-08eb1070e746" (UID: "49f715c7-a10b-4da9-9b21-08eb1070e746"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:46:43.688642 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.688564    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f715c7-a10b-4da9-9b21-08eb1070e746-kube-api-access-7hvv4" (OuterVolumeSpecName: "kube-api-access-7hvv4") pod "49f715c7-a10b-4da9-9b21-08eb1070e746" (UID: "49f715c7-a10b-4da9-9b21-08eb1070e746"). InnerVolumeSpecName "kube-api-access-7hvv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:46:43.787486 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.787453    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-dshm\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:46:43.787486 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.787480    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-tmp-dir\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:46:43.787635 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.787495    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hvv4\" (UniqueName: \"kubernetes.io/projected/49f715c7-a10b-4da9-9b21-08eb1070e746-kube-api-access-7hvv4\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:46:43.787635 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.787510    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-model-cache\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:46:43.787635 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.787523    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f715c7-a10b-4da9-9b21-08eb1070e746-home\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:46:43.787635 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.787545    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f715c7-a10b-4da9-9b21-08eb1070e746-tls-certs\") on node \"ip-10-0-135-243.ec2.internal\" DevicePath \"\""
Sep 17 16:46:43.904623 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.904600    2555 generic.go:356] "Generic (PLEG): container finished" podID="49f715c7-a10b-4da9-9b21-08eb1070e746" containerID="abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1" exitCode=0
Sep 17 16:46:43.904957 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.904643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6" event={"ID":"49f715c7-a10b-4da9-9b21-08eb1070e746","Type":"ContainerDied","Data":"abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1"}
Sep 17 16:46:43.904957 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.904679    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6" event={"ID":"49f715c7-a10b-4da9-9b21-08eb1070e746","Type":"ContainerDied","Data":"406cf2a76c67c3c0aaa2873f2093a7234651d4f513e413c2e309e5511461951e"}
Sep 17 16:46:43.904957 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.904700    2555 scope.go:117] "RemoveContainer" containerID="abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1"
Sep 17 16:46:43.904957 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.904646    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"
Sep 17 16:46:43.913688 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.913672    2555 scope.go:117] "RemoveContainer" containerID="abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1"
Sep 17 16:46:43.913961 ip-10-0-135-243 kubenswrapper[2555]: E0917 16:46:43.913942    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1\": container with ID starting with abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1 not found: ID does not exist" containerID="abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1"
Sep 17 16:46:43.914013 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.913969    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1"} err="failed to get container status \"abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1\": rpc error: code = NotFound desc = could not find container \"abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1\": container with ID starting with abf7633d96dbf304afe6c562878de49b324c501261c37873dc002e10cd53eae1 not found: ID does not exist"
Sep 17 16:46:43.927814 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.927795    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"]
Sep 17 16:46:43.932147 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:43.932126    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v5jl6"]
Sep 17 16:46:44.694146 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:46:44.694111    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49f715c7-a10b-4da9-9b21-08eb1070e746" path="/var/lib/kubelet/pods/49f715c7-a10b-4da9-9b21-08eb1070e746/volumes"
Sep 17 16:47:56.844272 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:47:56.844238    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-crjlt_7e9582a6-488e-486d-a7ab-20ad6fb18be2/istio-proxy/0.log"
Sep 17 16:47:56.849782 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:47:56.849754    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-crjlt_7e9582a6-488e-486d-a7ab-20ad6fb18be2/istio-proxy/1.log"
Sep 17 16:47:57.788252 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:47:57.788203    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-crjlt_7e9582a6-488e-486d-a7ab-20ad6fb18be2/istio-proxy/0.log"
Sep 17 16:47:57.794297 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:47:57.794263    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-crjlt_7e9582a6-488e-486d-a7ab-20ad6fb18be2/istio-proxy/1.log"
Sep 17 16:47:59.023115 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:47:59.023079    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-5j4ml_2a84d3db-8d6e-4554-914b-184e8777dea7/authorino/0.log"
Sep 17 16:48:04.897693 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:04.897653    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nlf6d_a6acb82a-8eb6-4508-9345-88b203e27b0e/global-pull-secret-syncer/0.log"
Sep 17 16:48:05.075000 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:05.074971    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p65h2_fd89a8d5-387c-4f46-80d2-1c7ad2bc8286/konnectivity-agent/0.log"
Sep 17 16:48:05.172808 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:05.172731    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-243.ec2.internal_8ce232a731178d99d36cb8d051eb3270/haproxy/0.log"
Sep 17 16:48:09.791429 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:09.791401    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-5j4ml_2a84d3db-8d6e-4554-914b-184e8777dea7/authorino/0.log"
Sep 17 16:48:11.855488 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:11.855453    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-cmkxg_1ca0910f-5836-46ac-9e2e-20fbb8d3f7a2/cluster-monitoring-operator/0.log"
Sep 17 16:48:12.033548 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:12.033510    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-gchxf_e4be7e13-5d73-4f12-8333-e4ae9dcac435/monitoring-plugin/0.log"
Sep 17 16:48:12.441030 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:12.440991    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqg8m_0b38eede-593c-46ba-9502-ee099ba94a1c/node-exporter/0.log"
Sep 17 16:48:12.548464 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:12.548427    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqg8m_0b38eede-593c-46ba-9502-ee099ba94a1c/kube-rbac-proxy/0.log"
Sep 17 16:48:12.623313 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:12.623288    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqg8m_0b38eede-593c-46ba-9502-ee099ba94a1c/init-textfile/0.log"
Sep 17 16:48:14.942085 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.942051    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"]
Sep 17 16:48:14.942467 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.942332    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f715c7-a10b-4da9-9b21-08eb1070e746" containerName="main"
Sep 17 16:48:14.942467 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.942357    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f715c7-a10b-4da9-9b21-08eb1070e746" containerName="main"
Sep 17 16:48:14.942467 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.942434    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f715c7-a10b-4da9-9b21-08eb1070e746" containerName="main"
Sep 17 16:48:14.945310 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.945283    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:14.952244 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.952224    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-przwd\"/\"kube-root-ca.crt\""
Sep 17 16:48:14.952656 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.952641    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-przwd\"/\"openshift-service-ca.crt\""
Sep 17 16:48:14.954538 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.954518    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-przwd\"/\"default-dockercfg-gx5hg\""
Sep 17 16:48:14.973152 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.973125    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"]
Sep 17 16:48:14.994202 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.994181    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-proc\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:14.994313 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.994213    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-sys\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:14.994313 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.994243    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-lib-modules\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:14.994313 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.994272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjtf\" (UniqueName: \"kubernetes.io/projected/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-kube-api-access-kpjtf\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:14.994491 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:14.994327    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-podres\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.095511 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.095487    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-proc\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.095599 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.095518    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-sys\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.095599 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.095546    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-lib-modules\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.095681 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.095585    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjtf\" (UniqueName: \"kubernetes.io/projected/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-kube-api-access-kpjtf\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.095681 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.095624    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-sys\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.095681 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.095629    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-podres\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.095681 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.095596    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-proc\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.095681 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.095662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-lib-modules\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.095870 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.095720    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-podres\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.107132 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.107107    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjtf\" (UniqueName: \"kubernetes.io/projected/b3f4ba11-ac72-47dc-8c82-43c38abccd4d-kube-api-access-kpjtf\") pod \"perf-node-gather-daemonset-2mrtv\" (UID: \"b3f4ba11-ac72-47dc-8c82-43c38abccd4d\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.254454 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.254393    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:15.532222 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:15.532087    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"]
Sep 17 16:48:15.534807 ip-10-0-135-243 kubenswrapper[2555]: W0917 16:48:15.534778    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb3f4ba11_ac72_47dc_8c82_43c38abccd4d.slice/crio-853fed408aca1dbd7e3bfed18e6a96048751e196f905423df721bcca6a39e9eb WatchSource:0}: Error finding container 853fed408aca1dbd7e3bfed18e6a96048751e196f905423df721bcca6a39e9eb: Status 404 returned error can't find the container with id 853fed408aca1dbd7e3bfed18e6a96048751e196f905423df721bcca6a39e9eb
Sep 17 16:48:16.349593 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:16.349560    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv" event={"ID":"b3f4ba11-ac72-47dc-8c82-43c38abccd4d","Type":"ContainerStarted","Data":"9fcf5a4e255e7ad4b87ed73cbc2756bb87ab56142b2120e4949d1545f9727092"}
Sep 17 16:48:16.350015 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:16.349609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv" event={"ID":"b3f4ba11-ac72-47dc-8c82-43c38abccd4d","Type":"ContainerStarted","Data":"853fed408aca1dbd7e3bfed18e6a96048751e196f905423df721bcca6a39e9eb"}
Sep 17 16:48:16.350015 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:16.349641    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:16.370164 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:16.370120    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv" podStartSLOduration=2.370106172 podStartE2EDuration="2.370106172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:48:16.368324501 +0000 UTC m=+4178.259185046" watchObservedRunningTime="2026-09-17 16:48:16.370106172 +0000 UTC m=+4178.260966751"
Sep 17 16:48:17.951243 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:17.951206    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p2dt6_914dd600-d7ac-49f2-b490-5a4d82d748ae/dns/0.log"
Sep 17 16:48:17.979546 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:17.979512    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p2dt6_914dd600-d7ac-49f2-b490-5a4d82d748ae/kube-rbac-proxy/0.log"
Sep 17 16:48:18.037020 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:18.036990    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rjjjt_6d538ca1-32a3-4bd3-95f4-44177e3d30ab/dns-node-resolver/0.log"
Sep 17 16:48:18.849530 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:18.849491    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4k758_33b10e70-61bd-4101-9a26-e0349b249102/node-ca/0.log"
Sep 17 16:48:20.049953 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:20.049920    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-crjlt_7e9582a6-488e-486d-a7ab-20ad6fb18be2/istio-proxy/0.log"
Sep 17 16:48:20.054768 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:20.054742    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-crjlt_7e9582a6-488e-486d-a7ab-20ad6fb18be2/istio-proxy/1.log"
Sep 17 16:48:20.721630 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:20.721599    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h5tw5_9addd732-ac2d-4f71-9057-def9b3c7fbe5/serve-healthcheck-canary/0.log"
Sep 17 16:48:21.488938 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:21.488904    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s6n66_18308130-5434-45b2-9167-32b5f771ea52/kube-rbac-proxy/0.log"
Sep 17 16:48:21.522205 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:21.522181    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s6n66_18308130-5434-45b2-9167-32b5f771ea52/exporter/0.log"
Sep 17 16:48:21.556903 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:21.556881    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s6n66_18308130-5434-45b2-9167-32b5f771ea52/extractor/0.log"
Sep 17 16:48:22.363102 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:22.363077    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-2mrtv"
Sep 17 16:48:25.043243 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:25.043214    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-crjlt_7e9582a6-488e-486d-a7ab-20ad6fb18be2/istio-proxy/0.log"
Sep 17 16:48:25.048180 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:25.048120    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-crjlt_7e9582a6-488e-486d-a7ab-20ad6fb18be2/istio-proxy/1.log"
Sep 17 16:48:25.186867 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:25.186840    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65f7746b69-pcjww_be93defc-ebcd-4c45-a09d-bfbc53de9f76/manager/0.log"
Sep 17 16:48:25.220748 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:25.220718    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jvkjc_2dc48b0e-1935-4164-8bfa-069634107495/openshift-lws-operator/0.log"
Sep 17 16:48:26.203365 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:26.203336    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-crjlt_7e9582a6-488e-486d-a7ab-20ad6fb18be2/istio-proxy/0.log"
Sep 17 16:48:26.208478 ip-10-0-135-243 kubenswrapper[2555]: I0917 16:48:26.208454    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-crjlt_7e9582a6-488e-486d-a7ab-20ad6fb18be2/istio-proxy/1.log"