Jul 06 20:34:46.971098 ip-10-0-131-45 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 20:34:47.261999 ip-10-0-131-45 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:34:47.261999 ip-10-0-131-45 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 20:34:47.261999 ip-10-0-131-45 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:34:47.261999 ip-10-0-131-45 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 20:34:47.261999 ip-10-0-131-45 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:34:47.263555 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.263473    2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 20:34:47.268674 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268652    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:34:47.268674 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268670    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:34:47.268674 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268675    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:34:47.268674 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268678    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:34:47.268674 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268681    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268684    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268687    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268690    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268693    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268696    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268699    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268701    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268704    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268708    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268710    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268713    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268715    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268718    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268721    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268724    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268726    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268729    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268732    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268734    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:34:47.268915 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268737    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268739    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268742    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268745    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268748    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268750    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268754    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268759    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268763    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268765    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268769    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268771    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268774    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268777    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268780    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268783    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268786    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268788    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268791    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268794    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:34:47.269430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268797    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268799    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268802    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268804    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268809    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268814    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268818    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268822    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268825    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268828    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268831    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268833    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268836    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268839    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268841    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268844    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268847    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268849    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268853    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:34:47.269973 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268856    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268859    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268862    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268865    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268867    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268870    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268873    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268894    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268899    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268903    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268907    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268911    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268914    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268917    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268919    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268923    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268927    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268930    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268933    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:34:47.270475 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268936    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268939    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268942    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268946    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268950    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268954    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268957    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268959    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268963    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268966    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268969    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268971    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268974    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268976    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268981    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268984    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.268988    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269709    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269716    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:34:47.270985 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269719    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269722    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269726    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269728    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269732    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269734    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269737    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269741    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269743    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269746    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269749    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269751    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269754    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269757    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269759    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269765    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269768    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269773    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269776    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:34:47.271463 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269780    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269783    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269786    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269789    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269792    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269795    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269798    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269801    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269803    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269807    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269809    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269812    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269815    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269817    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269821    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269824    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269826    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269829    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269833    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:34:47.272017 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269835    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269839    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269843    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269845    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269848    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269851    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269853    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269856    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269859    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269862    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269865    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269868    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269871    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269874    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269877    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269879    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269899    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269902    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269905    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:34:47.272511 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269908    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269912    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269914    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269917    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269920    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269923    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269925    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269928    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269931    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269934    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269939    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269943    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269946    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269949    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269951    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269954    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269957    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269959    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269962    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:34:47.273031 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269964    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269967    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269970    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269972    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269975    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269979    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269983    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269986    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269988    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269992    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269995    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.269998    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270001    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270004    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270007    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270009    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270012    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270015    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270018    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270020    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:34:47.273519 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270023    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270420    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270426    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270429    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270431    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270434    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270437    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270440    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270442    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270445    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270450    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270454    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270458    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270461    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270464    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270467    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270470    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270474    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270477    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:34:47.274059 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270480    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270483    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270486    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270489    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270492    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270495    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270498    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270501    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270504    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270507    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270510    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270513    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270515    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270518    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270521    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270523    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270526    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270530    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270534    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:34:47.274537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270537    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270540    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270543    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270546    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270549    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270552    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270554    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270557    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270560    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270562    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270565    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270569    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270572    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270575    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270578    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270580    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270583    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270586    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270589    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270591    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:34:47.275071 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270594    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270597    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270600    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270602    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270605    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270607    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270610    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270613    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270615    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270618    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270621    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270623    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270626    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270629    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270631    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270634    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270636    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270639    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270643    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270646    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:34:47.275585 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270648    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270651    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270654    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270657    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270660    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270663    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270665    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270668    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270671    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270674    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270678    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270681    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270684    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270687    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270690    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270692    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270695    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270698    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270701    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:34:47.276112 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270703    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270706    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.270708    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270782    2550 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270789    2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270796    2550 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270800    2550 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270804    2550 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270807    2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270811    2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270816    2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270819    2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270822    2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270828    2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270834    2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270837    2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270840    2550 flags.go:64] FLAG: --cgroup-root=""
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270843    2550 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270846    2550 flags.go:64] FLAG: --client-ca-file=""
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270850    2550 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270853    2550 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270858    2550 flags.go:64] FLAG: --cluster-domain=""
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270861    2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270864    2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 20:34:47.276598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270868    2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270871    2550 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270875    2550 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270878    2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270897    2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270901    2550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270905    2550 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270908    2550 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270911    2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270914    2550 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270917    2550 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270921    2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270924    2550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270927    2550 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270930    2550 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270934    2550 flags.go:64] FLAG: --enable-server="true"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270937    2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270942    2550 flags.go:64] FLAG: --event-burst="100"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270945    2550 flags.go:64] FLAG: --event-qps="50"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270948    2550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270951    2550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270954    2550 flags.go:64] FLAG: --eviction-hard=""
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270958    2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270961    2550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270964    2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 20:34:47.277238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270967    2550 flags.go:64] FLAG: --eviction-soft=""
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270970    2550 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270973    2550 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270977    2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270980    2550 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270983    2550 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270989    2550 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270992    2550 flags.go:64] FLAG: --feature-gates=""
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.270996    2550 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271000    2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271003    2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271006    2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271009    2550 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271012    2550 flags.go:64] FLAG: --help="false"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271015    2550 flags.go:64] FLAG: --hostname-override="ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271018    2550 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271021    2550 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271024    2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271028    2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271031    2550 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271034    2550 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271037    2550 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271040    2550 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 20:34:47.277940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271046    2550 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271049    2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271052    2550 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271055    2550 flags.go:64] FLAG: --kube-reserved=""
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271058    2550 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271061    2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271064    2550 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271067    2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271070    2550 flags.go:64] FLAG: --lock-file=""
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271073    2550 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271076    2550 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271080    2550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271085    2550 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271089    2550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271093    2550 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271101    2550 flags.go:64] FLAG: --logging-format="text"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271104    2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271108    2550 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271111    2550 flags.go:64] FLAG: --manifest-url=""
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271114    2550 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271118    2550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271121    2550 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271125    2550 flags.go:64] FLAG: --max-pods="110"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271128    2550 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271131    2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 20:34:47.278557 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271134    2550 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271137    2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271140    2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271143    2550 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271146    2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271154    2550 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271157    2550 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271162    2550 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271165    2550 flags.go:64] FLAG: --pod-cidr=""
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271168    2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271174    2550 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271177    2550 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271181    2550 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271184    2550 flags.go:64] FLAG: --port="10250"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271187    2550 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271190    2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09bbc21b2c4929de1"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271193    2550 flags.go:64] FLAG: --qos-reserved=""
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271196    2550 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271199    2550 flags.go:64] FLAG: --register-node="true"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271202    2550 flags.go:64] FLAG: --register-with-taints=""
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271206    2550 flags.go:64] FLAG: --registry-burst="10"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271216    2550 flags.go:64] FLAG: --registry-qps="5"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271220    2550 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271223    2550 flags.go:64] FLAG: --reserved-memory=""
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271227    2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 20:34:47.279276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271231    2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271234    2550 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271237    2550 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271240    2550 flags.go:64] FLAG: --runonce="false"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271243    2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271246    2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271249    2550 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271253    2550 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271256    2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271258    2550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271262    2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271265    2550 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271268    2550 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271271    2550 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271273    2550 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271278    2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271281    2550 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271284    2550 flags.go:64] FLAG: --system-cgroups=""
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271287    2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271293    2550 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271296    2550 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271299    2550 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271303    2550 flags.go:64] FLAG: --tls-min-version=""
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271306    2550 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271309    2550 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 20:34:47.279946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271312    2550 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271314    2550 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271317    2550 flags.go:64] FLAG: --v="2"
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271321    2550 flags.go:64] FLAG: --version="false"
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271327    2550 flags.go:64] FLAG: --vmodule=""
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271331    2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271334    2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271456    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271461    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271464    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271468    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271471    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271474    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271477    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271480    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271483    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271486    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271489    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271491    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271494    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271497    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:34:47.280595 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271499    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271504    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271506    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271509    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271512    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271514    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271517    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271520    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271523    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271525    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271528    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271531    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271534    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271536    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271539    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271543    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271546    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271549    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271553    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:34:47.281272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271555    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271558    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271561    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271564    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271566    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271569    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271572    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271574    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271577    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271580    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271582    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271585    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271588    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271590    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271594    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271628    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271631    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271634    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271637    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271640    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:34:47.281769 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271643    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271646    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271649    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271651    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271654    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271657    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271660    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271662    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271667    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271670    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271674    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271678    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271682    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271685    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271688    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271691    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271695    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271699    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271702    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:34:47.282306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271705    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271708    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271711    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271714    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271717    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271720    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271723    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271727    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271730    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271733    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271736    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271740    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271744    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271747    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271750    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271753    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271756    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271759    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271761    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271764    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:34:47.282800 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271767    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271772    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271775    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271778    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271782    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271784    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.271787    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.271797    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.278699    2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.278714    2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.279285    2550 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.279296    2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279362    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279368    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279371    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:34:47.283357 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279374    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279379    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279382    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279385    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279389    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279392    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279395    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279398    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279401    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279404    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279407    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279410    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279413    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279416    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279420    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279422    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279426    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279429    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279432    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279435    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:34:47.283759 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279438    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279442    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279445    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279448    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279451    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279454    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279457    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279460    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279463    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279466    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279469    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279472    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279476    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279480    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279483    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279486    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279489    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279492    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279495    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:34:47.284298 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279497    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279500    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279503    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279506    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279508    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279511    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279514    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279516    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279519    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279522    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279525    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279528    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279531    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279534    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279536    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279540    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279542    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279545    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279548    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:34:47.284784 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279552    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279555    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279557    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279560    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279563    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279566    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279568    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279571    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279574    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279577    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279579    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279582    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279585    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279587    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279590    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279593    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279596    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279598    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279601    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279604    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:34:47.285303 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279607    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279626    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279630    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279634    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279637    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279640    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279643    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279647    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279651    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279660    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279664    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279667    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279671    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279675    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279678    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279681    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279683    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:34:47.285826 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279686    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.279692    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279810    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279814    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279818    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279821    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279824    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279826    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279829    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279832    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279835    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279838    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279841    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279844    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279846    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:34:47.286306 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279849    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279852    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279854    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279857    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279861    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279865    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279867    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279870    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279873    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279875    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279879    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279894    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279896    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279899    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279902    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279905    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279908    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279910    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279913    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:34:47.286830 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279916    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279919    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279922    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279924    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279927    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279929    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279932    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279935    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279937    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279940    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279943    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279946    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279949    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279952    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279954    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279959    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279962    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279966    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279969    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279972    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:34:47.287398 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279975    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279978    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279980    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279983    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279986    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279989    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279992    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279995    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.279997    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280000    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280003    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280006    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280008    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280011    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280014    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280016    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280019    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280022    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280025    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280028    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:34:47.287952 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280031    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280033    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280036    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280039    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280041    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280044    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280047    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280052    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280056    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280060    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280064    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280069    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280073    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280078    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280081    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280091    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280094    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280102    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280106    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280109    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:34:47.288479 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280112    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280115    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280118    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280121    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280124    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280127    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:47.280130    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.280135    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.280239    2550 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.282208    2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.283391    2550 server.go:1019] "Starting client certificate rotation"
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.283502    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.283535    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:34:47.289032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.285505    2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:34:47.289424 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.286681    2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:34:47.297650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.297628    2550 log.go:25] "Validated CRI v1 runtime API"
Jul 06 20:34:47.302539 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.302518    2550 log.go:25] "Validated CRI v1 image API"
Jul 06 20:34:47.303454 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.303436    2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 20:34:47.305917 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.305874    2550 fs.go:135] Filesystem UUIDs: map[64077bfb-9347-44ed-bf4e-e53e2f4d47f0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9e7d367a-9f30-4d30-af34-892c80e3ac80:/dev/nvme0n1p3]
Jul 06 20:34:47.306002 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.305914    2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 20:34:47.311113 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.311003    2550 manager.go:216] Machine: {Timestamp:2026-07-06 20:34:47.309788045 +0000 UTC m=+0.251091886 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099373 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22bfa9a1d6c4908c795bf42e5c60e9 SystemUUID:ec22bfa9-a1d6-c490-8c79-5bf42e5c60e9 BootID:6e0088ac-2b43-436b-9761-333132d7c649 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:df:2d:91:53:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:df:2d:91:53:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:77:94:a6:73:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 20:34:47.311113 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.311111    2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 20:34:47.311226 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.311186    2550 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 20:34:47.313703 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.313679    2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 20:34:47.313843 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.313706    2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 20:34:47.314833 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.314822    2550 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 20:34:47.314869 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.314836    2550 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 20:34:47.314869 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.314844    2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:34:47.315354 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.315342    2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:34:47.316591 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.316580    2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 20:34:47.320516 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.320504    2550 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:34:47.320799 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.320788    2550 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 20:34:47.321174 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.321163    2550 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 20:34:47.321228 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.321184    2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 20:34:47.321228 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.321201    2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 20:34:47.321228 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.321210    2550 kubelet.go:398] "Adding apiserver pod source"
Jul 06 20:34:47.321228 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.321219    2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 20:34:47.321646 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.321633    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:34:47.321712 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.321653    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:34:47.323426 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.323409    2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 20:34:47.323682 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.323663    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:34:47.325231 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.325214    2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 20:34:47.325326 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.325255    2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 20:34:47.326291 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326275    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 20:34:47.326379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326297    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 20:34:47.326379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326308    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 20:34:47.326379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326317    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 20:34:47.326379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326330    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 20:34:47.326379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326339    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 20:34:47.326379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326352    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 20:34:47.326379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326362    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 20:34:47.326379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326373    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 20:34:47.326379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326382    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 20:34:47.326676 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326395    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 20:34:47.326676 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.326404    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 20:34:47.327029 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.327004    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 20:34:47.327029 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.327028    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 20:34:47.330505 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.330489    2550 server.go:1268] "Started kubelet"
Jul 06 20:34:47.330614 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.330578    2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 20:34:47.330746 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.330706    2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 20:34:47.330851 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.330771    2550 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 20:34:47.331712 ip-10-0-131-45 systemd[1]: Started Kubernetes Kubelet.
Jul 06 20:34:47.332583 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.332446    2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 20:34:47.333176 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.333160    2550 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 20:34:47.335273 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.335241    2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 20:34:47.335579 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.335550    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 20:34:47.335579 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.335564    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 20:34:47.337942 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.337921    2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 20:34:47.340210 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.340193    2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 20:34:47.340272 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.340213    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 20:34:47.340394 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.340375    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:47.340771 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.340751    2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 20:34:47.340885 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.340785    2550 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 20:34:47.340885 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.340034    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-45.ec2.internal.18bfcda54e63ad20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-45.ec2.internal,UID:ip-10-0-131-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-45.ec2.internal,},FirstTimestamp:2026-07-06 20:34:47.330458912 +0000 UTC m=+0.271762753,LastTimestamp:2026-07-06 20:34:47.330458912 +0000 UTC m=+0.271762753,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-45.ec2.internal,}"
Jul 06 20:34:47.340885 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.340800    2550 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 20:34:47.341059 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.340936    2550 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 20:34:47.341059 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.340945    2550 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 20:34:47.341161 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.341132    2550 factory.go:55] Registering systemd factory
Jul 06 20:34:47.341161 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.341152    2550 factory.go:223] Registration of the systemd container factory successfully
Jul 06 20:34:47.341417 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.341398    2550 factory.go:153] Registering CRI-O factory
Jul 06 20:34:47.341417 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.341419    2550 factory.go:223] Registration of the crio container factory successfully
Jul 06 20:34:47.341551 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.341473    2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 20:34:47.341551 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.341500    2550 factory.go:103] Registering Raw factory
Jul 06 20:34:47.341551 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.341515    2550 manager.go:1196] Started watching for new ooms in manager
Jul 06 20:34:47.341994 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.341977    2550 manager.go:349] Starting recovery of all containers
Jul 06 20:34:47.350630 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.350460    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 06 20:34:47.350714 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.350515    2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 06 20:34:47.353181 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.353164    2550 manager.go:354] Recovery completed
Jul 06 20:34:47.358353 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.358341    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:34:47.360343 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.360317    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:34:47.360419 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.360355    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:34:47.360419 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.360366    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:34:47.360953 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.360936    2550 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 20:34:47.360953 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.360952    2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 20:34:47.361103 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.360970    2550 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:34:47.361103 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.360972    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:34:47.362146 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.362128    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:34:47.362249 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.362160    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:34:47.362249 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.362188    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:34:47.363301 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.363288    2550 policy_none.go:49] "None policy: Start"
Jul 06 20:34:47.363366 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.363308    2550 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 20:34:47.363366 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.363325    2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 20:34:47.364062 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.363994    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-45.ec2.internal.18bfcda5502ba4ed  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-45.ec2.internal,UID:ip-10-0-131-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-45.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-45.ec2.internal,},FirstTimestamp:2026-07-06 20:34:47.360341229 +0000 UTC m=+0.301645094,LastTimestamp:2026-07-06 20:34:47.360341229 +0000 UTC m=+0.301645094,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-45.ec2.internal,}"
Jul 06 20:34:47.365473 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.365459    2550 policy_none.go:47] "Start"
Jul 06 20:34:47.376634 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.376570    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-45.ec2.internal.18bfcda5502beea8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-45.ec2.internal,UID:ip-10-0-131-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-45.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-45.ec2.internal,},FirstTimestamp:2026-07-06 20:34:47.360360104 +0000 UTC m=+0.301663945,LastTimestamp:2026-07-06 20:34:47.360360104 +0000 UTC m=+0.301663945,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-45.ec2.internal,}"
Jul 06 20:34:47.387600 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.387509    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-45.ec2.internal.18bfcda5502c15b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-45.ec2.internal,UID:ip-10-0-131-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-45.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-45.ec2.internal,},FirstTimestamp:2026-07-06 20:34:47.3603701 +0000 UTC m=+0.301673940,LastTimestamp:2026-07-06 20:34:47.3603701 +0000 UTC m=+0.301673940,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-45.ec2.internal,}"
Jul 06 20:34:47.390632 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.390572    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-825gh"
Jul 06 20:34:47.397440 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.397379    2550 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-45.ec2.internal.18bfcda5502ba4ed\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-45.ec2.internal.18bfcda5502ba4ed  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-45.ec2.internal,UID:ip-10-0-131-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-45.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-45.ec2.internal,},FirstTimestamp:2026-07-06 20:34:47.360341229 +0000 UTC m=+0.301645094,LastTimestamp:2026-07-06 20:34:47.362144431 +0000 UTC m=+0.303448279,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-45.ec2.internal,}"
Jul 06 20:34:47.398492 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.398479    2550 manager.go:337] "Starting Device Plugin manager"
Jul 06 20:34:47.398548 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.398510    2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 20:34:47.398548 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.398521    2550 server.go:87] "Starting device plugin registration server"
Jul 06 20:34:47.398763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.398750    2550 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 20:34:47.398828 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.398763    2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 20:34:47.398877 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.398852    2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 20:34:47.398957 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.398941    2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 20:34:47.399005 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.398959    2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 20:34:47.399407 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.399377    2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 20:34:47.399499 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.399430    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-825gh"
Jul 06 20:34:47.399499 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.399438    2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:47.466236 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.466207    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 20:34:47.469368 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.467417    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 20:34:47.469368 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.467463    2550 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 20:34:47.469368 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.467492    2550 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 20:34:47.469368 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.467532    2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 20:34:47.470519 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.470500    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:34:47.499238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.499216    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:34:47.500254 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.500231    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:34:47.500335 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.500263    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:34:47.500335 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.500274    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:34:47.500335 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.500296    2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.511593 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.511576    2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.511643 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.511597    2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-45.ec2.internal\": node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:47.555062 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.555037    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:47.567596 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.567575    2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal"]
Jul 06 20:34:47.567653 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.567645    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:34:47.568507 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.568491    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:34:47.568578 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.568519    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:34:47.568578 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.568529    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:34:47.571090 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.571077    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:34:47.571213 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.571200    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.571250 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.571230    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:34:47.571809 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.571793    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:34:47.571901 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.571845    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:34:47.571901 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.571858    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:34:47.571901 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.571861    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:34:47.571901 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.571879    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:34:47.572023 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.571910    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:34:47.574347 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.574333    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.574397 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.574357    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:34:47.575114 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.575096    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:34:47.575190 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.575127    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:34:47.575190 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.575137    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:34:47.594610 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.594588    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-45.ec2.internal\" not found" node="ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.599170 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.599153    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-45.ec2.internal\" not found" node="ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.642408 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.642332    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45a32833bf4d608b8ecac078de2241ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"45a32833bf4d608b8ecac078de2241ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.642408 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.642365    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45a32833bf4d608b8ecac078de2241ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"45a32833bf4d608b8ecac078de2241ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.642408 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.642383    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e825bc10f7326db6e76e5c99478ee31-config\") pod \"kube-apiserver-proxy-ip-10-0-131-45.ec2.internal\" (UID: \"3e825bc10f7326db6e76e5c99478ee31\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.655185 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.655162    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:47.743060 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.743035    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45a32833bf4d608b8ecac078de2241ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"45a32833bf4d608b8ecac078de2241ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.743154 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.743064    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45a32833bf4d608b8ecac078de2241ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"45a32833bf4d608b8ecac078de2241ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.743154 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.743082    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e825bc10f7326db6e76e5c99478ee31-config\") pod \"kube-apiserver-proxy-ip-10-0-131-45.ec2.internal\" (UID: \"3e825bc10f7326db6e76e5c99478ee31\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.743154 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.743137    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e825bc10f7326db6e76e5c99478ee31-config\") pod \"kube-apiserver-proxy-ip-10-0-131-45.ec2.internal\" (UID: \"3e825bc10f7326db6e76e5c99478ee31\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.743280 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.743165    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45a32833bf4d608b8ecac078de2241ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"45a32833bf4d608b8ecac078de2241ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.743280 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.743168    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45a32833bf4d608b8ecac078de2241ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"45a32833bf4d608b8ecac078de2241ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.756082 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.756059    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:47.856247 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.856208    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:47.898260 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.898187    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.902065 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:47.902033    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:47.957310 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:47.957279    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:48.058374 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:48.058326    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:48.158966 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:48.158883    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:48.260095 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:48.260064    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:48.283992 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.283972    2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 20:34:48.284543 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.284149    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:34:48.360643 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:48.360618    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found"
Jul 06 20:34:48.403036 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.403002    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:29:47 +0000 UTC" deadline="2028-01-15 15:07:24.470188188 +0000 UTC"
Jul 06 20:34:48.403036 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.403034    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13386h32m36.067158535s"
Jul 06 20:34:48.433259 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.433210    2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:34:48.441199 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.441177    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:48.452664 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.452635    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:34:48.454023 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.454008    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal"
Jul 06 20:34:48.468304 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.468280    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:34:48.632856 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:48.632824    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e825bc10f7326db6e76e5c99478ee31.slice/crio-3c79aadb1c90104a23b3c9e86bceefbc1d83dfbe300a283179983b5cde8acc06 WatchSource:0}: Error finding container 3c79aadb1c90104a23b3c9e86bceefbc1d83dfbe300a283179983b5cde8acc06: Status 404 returned error can't find the container with id 3c79aadb1c90104a23b3c9e86bceefbc1d83dfbe300a283179983b5cde8acc06
Jul 06 20:34:48.641762 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.641744    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:34:48.648960 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:48.648937    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a32833bf4d608b8ecac078de2241ac.slice/crio-2390fbd6e4ac86eda1228653f030965f7316088968bc44fbe1ead45b77f1f6e3 WatchSource:0}: Error finding container 2390fbd6e4ac86eda1228653f030965f7316088968bc44fbe1ead45b77f1f6e3: Status 404 returned error can't find the container with id 2390fbd6e4ac86eda1228653f030965f7316088968bc44fbe1ead45b77f1f6e3
Jul 06 20:34:48.746806 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.746722    2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:34:48.929141 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:48.929111    2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:34:49.322639 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.322373    2550 apiserver.go:52] "Watching apiserver"
Jul 06 20:34:49.330156 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.330129    2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 20:34:49.331548 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.331521    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-dnd82","kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7","openshift-cluster-node-tuning-operator/tuned-m7rjb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal","openshift-multus/multus-additional-cni-plugins-jn5nv","openshift-multus/network-metrics-daemon-6vs4n","openshift-network-diagnostics/network-check-target-kkslg","kube-system/konnectivity-agent-zbdwk","openshift-dns/node-resolver-fp74t","openshift-image-registry/node-ca-mqszb","openshift-multus/multus-lgsmp","openshift-network-operator/iptables-alerter-lqxtl","openshift-ovn-kubernetes/ovnkube-node-v7kmn"]
Jul 06 20:34:49.334673 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.334370    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:49.334673 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.334454    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:34:49.337153 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.337123    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.343254 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.343230    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.343361 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.343319    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 20:34:49.345178 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.345154    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.350314 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.349281    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:49.350314 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.349369    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:34:49.350851 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.350678    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 20:34:49.350954 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.350892    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dphrb\""
Jul 06 20:34:49.350954 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.350899    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 20:34:49.351220 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.351070    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:34:49.352102 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.351968    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:49.352102 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.352027    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:34:49.354538 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354433    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:34:49.354632 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354546    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-sysconfig\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.354632 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354574    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-lib-modules\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.354632 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354599    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-var-lib-kubelet\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.354632 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354623    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-tmp\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.354857 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354648    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-cnibin\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.354857 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354682    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.354857 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354709    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-kubernetes\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.354857 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354733    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-sysctl-conf\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.354857 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354760    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.354857 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354785    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:49.354857 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354809    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-socket-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.354857 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354851    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-host\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354878    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-system-cni-dir\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354902    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-os-release\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354926    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354953    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-cni-binary-copy\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.354980    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbt9t\" (UniqueName: \"kubernetes.io/projected/9866e3e7-f454-42b7-add8-3bcf2b9f797a-kube-api-access-jbt9t\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355007    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355034    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-systemd\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355056    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-run\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355081    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cd6de51-6683-4b89-b584-0cb22a3b8833-kubelet-config\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355115    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cd6de51-6683-4b89-b584-0cb22a3b8833-dbus\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355142    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355168    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-registration-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355193    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-sysctl-d\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355218    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.355281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355272    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-device-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.356035 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355375    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-tuned\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.356035 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355409    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2bs\" (UniqueName: \"kubernetes.io/projected/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-kube-api-access-pf2bs\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.356035 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355563    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-sys-fs\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.356035 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355619    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-modprobe-d\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.356035 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355676    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-sys\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.356035 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355711    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v7jv\" (UniqueName: \"kubernetes.io/projected/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-kube-api-access-5v7jv\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.356035 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.355737    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9hd\" (UniqueName: \"kubernetes.io/projected/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-kube-api-access-cq9hd\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.356966 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.356945    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 20:34:49.357163 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.357143    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 20:34:49.357245 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.357204    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 20:34:49.357433 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.357416    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 20:34:49.357612 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.357594    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 20:34:49.357701 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.357682    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rxvf9\""
Jul 06 20:34:49.357830 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.357816    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mx9gf\""
Jul 06 20:34:49.357932 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.357899    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 20:34:49.358043 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.358013    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 20:34:49.358105 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.358077    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 20:34:49.358558 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.358539    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-h9vkc\""
Jul 06 20:34:49.358685 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.358669    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 20:34:49.358848 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.358833    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.360283 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.359862    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.361155 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.361135    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 20:34:49.361371 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.361354    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bgbdp\""
Jul 06 20:34:49.361682 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.361662    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 20:34:49.362481 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.362459    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 20:34:49.362689 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.362671    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 20:34:49.363137 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.363116    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-588jl\""
Jul 06 20:34:49.363397 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.363378    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 20:34:49.364514 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.363689    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.364514 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.364278    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:34:49.366623 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.365294    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.366623 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.366419    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 20:34:49.366623 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.366485    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-66qlw\""
Jul 06 20:34:49.368302 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.367604    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:34:49.368302 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.367819    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.368302 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.368147    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 20:34:49.369250 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.368695    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47pwf\""
Jul 06 20:34:49.369250 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.368726    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 20:34:49.374981 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.374626    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 20:34:49.375079 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.375048    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7l8k6\""
Jul 06 20:34:49.377302 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.377268    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 20:34:49.377396 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.377321    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 20:34:49.377467 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.377270    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 20:34:49.377538 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.377517    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 20:34:49.377658 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.377621    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 20:34:49.403544 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.403524    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k97tz"
Jul 06 20:34:49.409574 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.409548    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k97tz"
Jul 06 20:34:49.442580 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.442560    2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 20:34:49.456501 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456470    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-run-netns\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.456625 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456508    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-hostroot\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.456625 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456540    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-sysctl-d\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.456625 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456567    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.456625 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456591    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-run-systemd\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.456625 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456615    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-run-ovn-kubernetes\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456638    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6fe996a8-9939-4be3-a9d2-a574dbc4ce24-serviceca\") pod \"node-ca-mqszb\" (UID: \"6fe996a8-9939-4be3-a9d2-a574dbc4ce24\") " pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456663    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4f4c0df-6488-42fa-b247-34d41d7284a7-host-slash\") pod \"iptables-alerter-lqxtl\" (UID: \"e4f4c0df-6488-42fa-b247-34d41d7284a7\") " pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456691    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2bs\" (UniqueName: \"kubernetes.io/projected/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-kube-api-access-pf2bs\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456717    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-modprobe-d\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456743    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-sys\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456766    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-run-netns\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456790    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-etc-openvswitch\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456815    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-run-k8s-cni-cncf-io\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456840    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-var-lib-kubelet\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456881    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-sysconfig\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456905    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-tmp\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.456934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456930    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-cnibin\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456953    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-run-ovn\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.456982    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457006    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjpvv\" (UniqueName: \"kubernetes.io/projected/77f30a43-0971-4ca0-af13-9baba5202359-kube-api-access-gjpvv\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457031    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-host\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457056    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-system-cni-dir\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457091    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457151    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-systemd-units\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457188    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-node-log\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457213    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-cni-bin\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457241    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-cni-netd\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457265    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19194284-20fc-4878-8adc-3e77e8cc918c-ovnkube-config\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457268    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-cnibin\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457297    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-cni-binary-copy\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457328    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbt9t\" (UniqueName: \"kubernetes.io/projected/9866e3e7-f454-42b7-add8-3bcf2b9f797a-kube-api-access-jbt9t\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457353    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457380    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248l8\" (UniqueName: \"kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8\") pod \"network-check-target-kkslg\" (UID: \"7510d9cd-07f1-4c52-ad19-3a31394c9300\") " pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:49.457504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457406    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-os-release\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457454    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-systemd\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457457    2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457499    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-run\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457525    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cd6de51-6683-4b89-b584-0cb22a3b8833-dbus\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457566    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-run-openvswitch\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457594    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-system-cni-dir\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457614    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-cnibin\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457636    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-multus-conf-dir\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457664    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-device-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457693    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tnjv\" (UniqueName: \"kubernetes.io/projected/19194284-20fc-4878-8adc-3e77e8cc918c-kube-api-access-5tnjv\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457722    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5eef1e47-1131-4820-b054-5ad627772e0f-hosts-file\") pod \"node-resolver-fp74t\" (UID: \"5eef1e47-1131-4820-b054-5ad627772e0f\") " pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457758    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eef1e47-1131-4820-b054-5ad627772e0f-tmp-dir\") pod \"node-resolver-fp74t\" (UID: \"5eef1e47-1131-4820-b054-5ad627772e0f\") " pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457783    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpsm\" (UniqueName: \"kubernetes.io/projected/6fe996a8-9939-4be3-a9d2-a574dbc4ce24-kube-api-access-vzpsm\") pod \"node-ca-mqszb\" (UID: \"6fe996a8-9939-4be3-a9d2-a574dbc4ce24\") " pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457809    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f30a43-0971-4ca0-af13-9baba5202359-cni-binary-copy\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457832    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f30a43-0971-4ca0-af13-9baba5202359-multus-daemon-config\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457911    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457858    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-tuned\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.458508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458025    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-host\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458038    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-sysconfig\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457150    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-sys\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.457212    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458288    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-run\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458330    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-cni-binary-copy\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458344    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458397    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-modprobe-d\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458418    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-device-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458501    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cd6de51-6683-4b89-b584-0cb22a3b8833-dbus\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458538    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458545    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-sys-fs\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458578    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-var-lib-openvswitch\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458598    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-systemd\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458620    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f090afa-8dff-4167-b753-523344760615-konnectivity-ca\") pod \"konnectivity-agent-zbdwk\" (UID: \"8f090afa-8dff-4167-b753-523344760615\") " pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458635    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-sysctl-d\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458678    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-sys-fs\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.459333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458726    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn77j\" (UniqueName: \"kubernetes.io/projected/5eef1e47-1131-4820-b054-5ad627772e0f-kube-api-access-nn77j\") pod \"node-resolver-fp74t\" (UID: \"5eef1e47-1131-4820-b054-5ad627772e0f\") " pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458757    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-multus-socket-dir-parent\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458800    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v7jv\" (UniqueName: \"kubernetes.io/projected/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-kube-api-access-5v7jv\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458801    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-system-cni-dir\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458856    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9hd\" (UniqueName: \"kubernetes.io/projected/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-kube-api-access-cq9hd\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.458893    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-var-lib-cni-multus\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459017    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-lib-modules\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459056    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-var-lib-kubelet\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459127    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459159    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-var-lib-kubelet\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459168    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-lib-modules\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459202    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459246    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-var-lib-cni-bin\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459284    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-etc-kubernetes\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-kubernetes\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459338    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-sysctl-conf\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459377    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:49.459961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459440    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-kubernetes\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459449    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-socket-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459476    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-kubelet\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.459522    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459527    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19194284-20fc-4878-8adc-3e77e8cc918c-ovnkube-script-lib\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459554    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-run-multus-certs\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459587    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19194284-20fc-4878-8adc-3e77e8cc918c-ovn-node-metrics-cert\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459633    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459640    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-os-release\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.459679    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs podName:9866e3e7-f454-42b7-add8-3bcf2b9f797a nodeName:}" failed. No retries permitted until 2026-07-06 20:34:49.959640384 +0000 UTC m=+2.900944238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs") pod "network-metrics-daemon-6vs4n" (UID: "9866e3e7-f454-42b7-add8-3bcf2b9f797a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459706    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-os-release\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459711    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-slash\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459757    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-socket-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459800    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-log-socket\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459821    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-sysctl-conf\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459827    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f090afa-8dff-4167-b753-523344760615-agent-certs\") pod \"konnectivity-agent-zbdwk\" (UID: \"8f090afa-8dff-4167-b753-523344760615\") " pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:34:49.460945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459887    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-multus-cni-dir\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459927    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4f4c0df-6488-42fa-b247-34d41d7284a7-iptables-alerter-script\") pod \"iptables-alerter-lqxtl\" (UID: \"e4f4c0df-6488-42fa-b247-34d41d7284a7\") " pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.459974    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19194284-20fc-4878-8adc-3e77e8cc918c-env-overrides\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.460002    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc94p\" (UniqueName: \"kubernetes.io/projected/e4f4c0df-6488-42fa-b247-34d41d7284a7-kube-api-access-wc94p\") pod \"iptables-alerter-lqxtl\" (UID: \"e4f4c0df-6488-42fa-b247-34d41d7284a7\") " pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.460040    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cd6de51-6683-4b89-b584-0cb22a3b8833-kubelet-config\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.460105    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cd6de51-6683-4b89-b584-0cb22a3b8833-kubelet-config\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.460145    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.460172    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-registration-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.460199    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fe996a8-9939-4be3-a9d2-a574dbc4ce24-host\") pod \"node-ca-mqszb\" (UID: \"6fe996a8-9939-4be3-a9d2-a574dbc4ce24\") " pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.460279    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.460298    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-registration-dir\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.461802 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.460355    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret podName:1cd6de51-6683-4b89-b584-0cb22a3b8833 nodeName:}" failed. No retries permitted until 2026-07-06 20:34:49.960338171 +0000 UTC m=+2.901642016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret") pod "global-pull-secret-syncer-dnd82" (UID: "1cd6de51-6683-4b89-b584-0cb22a3b8833") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:34:49.463083 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.463059    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-etc-tuned\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.463193 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.463092    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-tmp\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.472786 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.472705    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2bs\" (UniqueName: \"kubernetes.io/projected/5d093092-ba3b-4059-ae54-cb0a6e2e4caf-kube-api-access-pf2bs\") pod \"multus-additional-cni-plugins-jn5nv\" (UID: \"5d093092-ba3b-4059-ae54-cb0a6e2e4caf\") " pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.472903 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.472819    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbt9t\" (UniqueName: \"kubernetes.io/projected/9866e3e7-f454-42b7-add8-3bcf2b9f797a-kube-api-access-jbt9t\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:49.473079 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.473057    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9hd\" (UniqueName: \"kubernetes.io/projected/7ef4bba0-0c6e-499f-a5d2-7eae1968e21f-kube-api-access-cq9hd\") pod \"aws-ebs-csi-driver-node-zl2j7\" (UID: \"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.473238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.473213    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v7jv\" (UniqueName: \"kubernetes.io/projected/ea9353a6-63d4-4119-bcef-6cb39dfb6b2e-kube-api-access-5v7jv\") pod \"tuned-m7rjb\" (UID: \"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.473733 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.473652    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" event={"ID":"3e825bc10f7326db6e76e5c99478ee31","Type":"ContainerStarted","Data":"3c79aadb1c90104a23b3c9e86bceefbc1d83dfbe300a283179983b5cde8acc06"}
Jul 06 20:34:49.474798 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.474755    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" event={"ID":"45a32833bf4d608b8ecac078de2241ac","Type":"ContainerStarted","Data":"2390fbd6e4ac86eda1228653f030965f7316088968bc44fbe1ead45b77f1f6e3"}
Jul 06 20:34:49.496121 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.495966    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:34:49.560881 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.560844    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-var-lib-cni-multus\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561059 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.560890    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561059 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.560921    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-var-lib-cni-bin\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561059 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.560946    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-etc-kubernetes\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561059 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.560980    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561059 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.560991    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-kubelet\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561059 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.560945    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-var-lib-cni-multus\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561059 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561035    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-etc-kubernetes\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561059 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561029    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19194284-20fc-4878-8adc-3e77e8cc918c-ovnkube-script-lib\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561045    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-var-lib-cni-bin\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561069    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-kubelet\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561094    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-run-multus-certs\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561126    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-run-multus-certs\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561136    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19194284-20fc-4878-8adc-3e77e8cc918c-ovn-node-metrics-cert\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561178    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-slash\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561205    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-log-socket\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561251    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-log-socket\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561258    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-slash\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561286    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f090afa-8dff-4167-b753-523344760615-agent-certs\") pod \"konnectivity-agent-zbdwk\" (UID: \"8f090afa-8dff-4167-b753-523344760615\") " pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561304    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-multus-cni-dir\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561321    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4f4c0df-6488-42fa-b247-34d41d7284a7-iptables-alerter-script\") pod \"iptables-alerter-lqxtl\" (UID: \"e4f4c0df-6488-42fa-b247-34d41d7284a7\") " pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561343    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19194284-20fc-4878-8adc-3e77e8cc918c-env-overrides\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561368    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc94p\" (UniqueName: \"kubernetes.io/projected/e4f4c0df-6488-42fa-b247-34d41d7284a7-kube-api-access-wc94p\") pod \"iptables-alerter-lqxtl\" (UID: \"e4f4c0df-6488-42fa-b247-34d41d7284a7\") " pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561399    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-multus-cni-dir\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561407    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fe996a8-9939-4be3-a9d2-a574dbc4ce24-host\") pod \"node-ca-mqszb\" (UID: \"6fe996a8-9939-4be3-a9d2-a574dbc4ce24\") " pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561431    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-run-netns\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.561459 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561454    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-hostroot\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561478    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-run-systemd\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561528    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-run-ovn-kubernetes\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561550    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6fe996a8-9939-4be3-a9d2-a574dbc4ce24-serviceca\") pod \"node-ca-mqszb\" (UID: \"6fe996a8-9939-4be3-a9d2-a574dbc4ce24\") " pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561577    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4f4c0df-6488-42fa-b247-34d41d7284a7-host-slash\") pod \"iptables-alerter-lqxtl\" (UID: \"e4f4c0df-6488-42fa-b247-34d41d7284a7\") " pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561606    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-run-netns\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561631    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-etc-openvswitch\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561657    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-run-k8s-cni-cncf-io\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561686    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-var-lib-kubelet\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561706    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4f4c0df-6488-42fa-b247-34d41d7284a7-host-slash\") pod \"iptables-alerter-lqxtl\" (UID: \"e4f4c0df-6488-42fa-b247-34d41d7284a7\") " pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561753    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19194284-20fc-4878-8adc-3e77e8cc918c-ovnkube-script-lib\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561764    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-run-systemd\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561655    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-run-netns\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561763    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-run-k8s-cni-cncf-io\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561716    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-run-ovn\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561768    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-etc-openvswitch\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561705    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-run-ovn-kubernetes\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561823    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-run-netns\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.562367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561853    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-run-ovn\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561878    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-hostroot\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561929    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjpvv\" (UniqueName: \"kubernetes.io/projected/77f30a43-0971-4ca0-af13-9baba5202359-kube-api-access-gjpvv\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.561992    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-host-var-lib-kubelet\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562029    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fe996a8-9939-4be3-a9d2-a574dbc4ce24-host\") pod \"node-ca-mqszb\" (UID: \"6fe996a8-9939-4be3-a9d2-a574dbc4ce24\") " pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562226    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-systemd-units\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562250    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4f4c0df-6488-42fa-b247-34d41d7284a7-iptables-alerter-script\") pod \"iptables-alerter-lqxtl\" (UID: \"e4f4c0df-6488-42fa-b247-34d41d7284a7\") " pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562268    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6fe996a8-9939-4be3-a9d2-a574dbc4ce24-serviceca\") pod \"node-ca-mqszb\" (UID: \"6fe996a8-9939-4be3-a9d2-a574dbc4ce24\") " pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562274    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-systemd-units\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562279    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19194284-20fc-4878-8adc-3e77e8cc918c-env-overrides\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562275    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-node-log\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562311    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-node-log\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562330    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-cni-bin\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562356    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-cni-netd\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562384    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19194284-20fc-4878-8adc-3e77e8cc918c-ovnkube-config\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562392    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-cni-bin\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562416    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-host-cni-netd\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562416    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248l8\" (UniqueName: \"kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8\") pod \"network-check-target-kkslg\" (UID: \"7510d9cd-07f1-4c52-ad19-3a31394c9300\") " pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:49.563281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562460    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-os-release\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562484    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-run-openvswitch\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562501    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-system-cni-dir\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562519    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-cnibin\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562541    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-multus-conf-dir\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562563    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tnjv\" (UniqueName: \"kubernetes.io/projected/19194284-20fc-4878-8adc-3e77e8cc918c-kube-api-access-5tnjv\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562594    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5eef1e47-1131-4820-b054-5ad627772e0f-hosts-file\") pod \"node-resolver-fp74t\" (UID: \"5eef1e47-1131-4820-b054-5ad627772e0f\") " pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562588    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-os-release\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562620    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eef1e47-1131-4820-b054-5ad627772e0f-tmp-dir\") pod \"node-resolver-fp74t\" (UID: \"5eef1e47-1131-4820-b054-5ad627772e0f\") " pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562645    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpsm\" (UniqueName: \"kubernetes.io/projected/6fe996a8-9939-4be3-a9d2-a574dbc4ce24-kube-api-access-vzpsm\") pod \"node-ca-mqszb\" (UID: \"6fe996a8-9939-4be3-a9d2-a574dbc4ce24\") " pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562672    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-cnibin\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562677    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f30a43-0971-4ca0-af13-9baba5202359-cni-binary-copy\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562717    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f30a43-0971-4ca0-af13-9baba5202359-multus-daemon-config\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562745    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-var-lib-openvswitch\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562786    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f090afa-8dff-4167-b753-523344760615-konnectivity-ca\") pod \"konnectivity-agent-zbdwk\" (UID: \"8f090afa-8dff-4167-b753-523344760615\") " pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562815    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn77j\" (UniqueName: \"kubernetes.io/projected/5eef1e47-1131-4820-b054-5ad627772e0f-kube-api-access-nn77j\") pod \"node-resolver-fp74t\" (UID: \"5eef1e47-1131-4820-b054-5ad627772e0f\") " pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562859    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-multus-socket-dir-parent\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562872    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-multus-conf-dir\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.564267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.562948    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19194284-20fc-4878-8adc-3e77e8cc918c-ovnkube-config\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.565127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.563008    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-var-lib-openvswitch\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.565127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.563016    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5eef1e47-1131-4820-b054-5ad627772e0f-hosts-file\") pod \"node-resolver-fp74t\" (UID: \"5eef1e47-1131-4820-b054-5ad627772e0f\") " pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.565127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.563104    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19194284-20fc-4878-8adc-3e77e8cc918c-run-openvswitch\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.565127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.563119    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-system-cni-dir\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.565127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.563164    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f30a43-0971-4ca0-af13-9baba5202359-multus-socket-dir-parent\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.565127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.563240    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f30a43-0971-4ca0-af13-9baba5202359-cni-binary-copy\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.565127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.563585    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eef1e47-1131-4820-b054-5ad627772e0f-tmp-dir\") pod \"node-resolver-fp74t\" (UID: \"5eef1e47-1131-4820-b054-5ad627772e0f\") " pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.565127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.563604    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f090afa-8dff-4167-b753-523344760615-konnectivity-ca\") pod \"konnectivity-agent-zbdwk\" (UID: \"8f090afa-8dff-4167-b753-523344760615\") " pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:34:49.565127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.563641    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f30a43-0971-4ca0-af13-9baba5202359-multus-daemon-config\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.565127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.564402    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f090afa-8dff-4167-b753-523344760615-agent-certs\") pod \"konnectivity-agent-zbdwk\" (UID: \"8f090afa-8dff-4167-b753-523344760615\") " pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:34:49.565847 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.565814    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19194284-20fc-4878-8adc-3e77e8cc918c-ovn-node-metrics-cert\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.580462 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.580394    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjpvv\" (UniqueName: \"kubernetes.io/projected/77f30a43-0971-4ca0-af13-9baba5202359-kube-api-access-gjpvv\") pod \"multus-lgsmp\" (UID: \"77f30a43-0971-4ca0-af13-9baba5202359\") " pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.581082 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.580901    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:34:49.581082 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.580932    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:34:49.581082 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.580949    2550 projected.go:196] Error preparing data for projected volume kube-api-access-248l8 for pod openshift-network-diagnostics/network-check-target-kkslg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:34:49.581082 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.581016    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8 podName:7510d9cd-07f1-4c52-ad19-3a31394c9300 nodeName:}" failed. No retries permitted until 2026-07-06 20:34:50.080996303 +0000 UTC m=+3.022300136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-248l8" (UniqueName: "kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8") pod "network-check-target-kkslg" (UID: "7510d9cd-07f1-4c52-ad19-3a31394c9300") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:34:49.583692 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.583662    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc94p\" (UniqueName: \"kubernetes.io/projected/e4f4c0df-6488-42fa-b247-34d41d7284a7-kube-api-access-wc94p\") pod \"iptables-alerter-lqxtl\" (UID: \"e4f4c0df-6488-42fa-b247-34d41d7284a7\") " pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.584493 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.584465    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tnjv\" (UniqueName: \"kubernetes.io/projected/19194284-20fc-4878-8adc-3e77e8cc918c-kube-api-access-5tnjv\") pod \"ovnkube-node-v7kmn\" (UID: \"19194284-20fc-4878-8adc-3e77e8cc918c\") " pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.584590 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.584533    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpsm\" (UniqueName: \"kubernetes.io/projected/6fe996a8-9939-4be3-a9d2-a574dbc4ce24-kube-api-access-vzpsm\") pod \"node-ca-mqszb\" (UID: \"6fe996a8-9939-4be3-a9d2-a574dbc4ce24\") " pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.584673 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.584584    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn77j\" (UniqueName: \"kubernetes.io/projected/5eef1e47-1131-4820-b054-5ad627772e0f-kube-api-access-nn77j\") pod \"node-resolver-fp74t\" (UID: \"5eef1e47-1131-4820-b054-5ad627772e0f\") " pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.659688 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.659652    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7"
Jul 06 20:34:49.670166 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.670137    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m7rjb"
Jul 06 20:34:49.683847 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.683827    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jn5nv"
Jul 06 20:34:49.692788 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.692765    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:34:49.703256 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.703237    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fp74t"
Jul 06 20:34:49.711091 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.711062    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mqszb"
Jul 06 20:34:49.720848 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.720829    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lgsmp"
Jul 06 20:34:49.729669 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.729649    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lqxtl"
Jul 06 20:34:49.737724 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.737703    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:34:49.966578 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.966489    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:49.966578 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:49.966553    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:49.966776 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.966646    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:34:49.966776 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.966717    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret podName:1cd6de51-6683-4b89-b584-0cb22a3b8833 nodeName:}" failed. No retries permitted until 2026-07-06 20:34:50.966695 +0000 UTC m=+3.907998830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret") pod "global-pull-secret-syncer-dnd82" (UID: "1cd6de51-6683-4b89-b584-0cb22a3b8833") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:34:49.966776 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.966648    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:34:49.966923 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:49.966810    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs podName:9866e3e7-f454-42b7-add8-3bcf2b9f797a nodeName:}" failed. No retries permitted until 2026-07-06 20:34:50.966794052 +0000 UTC m=+3.908097894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs") pod "network-metrics-daemon-6vs4n" (UID: "9866e3e7-f454-42b7-add8-3bcf2b9f797a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:34:50.167774 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:50.167735    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248l8\" (UniqueName: \"kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8\") pod \"network-check-target-kkslg\" (UID: \"7510d9cd-07f1-4c52-ad19-3a31394c9300\") " pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:50.167951 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:50.167895    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:34:50.167951 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:50.167911    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:34:50.167951 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:50.167921    2550 projected.go:196] Error preparing data for projected volume kube-api-access-248l8 for pod openshift-network-diagnostics/network-check-target-kkslg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:34:50.168094 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:50.167978    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8 podName:7510d9cd-07f1-4c52-ad19-3a31394c9300 nodeName:}" failed. No retries permitted until 2026-07-06 20:34:51.1679594 +0000 UTC m=+4.109263248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-248l8" (UniqueName: "kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8") pod "network-check-target-kkslg" (UID: "7510d9cd-07f1-4c52-ad19-3a31394c9300") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:34:50.358790 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:50.358746    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef4bba0_0c6e_499f_a5d2_7eae1968e21f.slice/crio-1f5c71c861c53aee5a285d980dc593f3bc4dbac56aa129bf321fa7fea652d820 WatchSource:0}: Error finding container 1f5c71c861c53aee5a285d980dc593f3bc4dbac56aa129bf321fa7fea652d820: Status 404 returned error can't find the container with id 1f5c71c861c53aee5a285d980dc593f3bc4dbac56aa129bf321fa7fea652d820
Jul 06 20:34:50.413512 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:50.413452    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:29:49 +0000 UTC" deadline="2028-01-24 23:07:28.190398293 +0000 UTC"
Jul 06 20:34:50.413512 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:50.413505    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13610h32m37.776898279s"
Jul 06 20:34:50.477598 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:50.477549    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7" event={"ID":"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f","Type":"ContainerStarted","Data":"1f5c71c861c53aee5a285d980dc593f3bc4dbac56aa129bf321fa7fea652d820"}
Jul 06 20:34:50.761772 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:50.761735    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f090afa_8dff_4167_b753_523344760615.slice/crio-0fa6cfa6c2b1df5c953bca615569910362acf83286c1780c970985f3eb071a49 WatchSource:0}: Error finding container 0fa6cfa6c2b1df5c953bca615569910362acf83286c1780c970985f3eb071a49: Status 404 returned error can't find the container with id 0fa6cfa6c2b1df5c953bca615569910362acf83286c1780c970985f3eb071a49
Jul 06 20:34:50.764490 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:50.764244    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eef1e47_1131_4820_b054_5ad627772e0f.slice/crio-373699f3ae8ca8ae3aad1a481395b2ff3c4225987a596eb8f828cfcbb52766ad WatchSource:0}: Error finding container 373699f3ae8ca8ae3aad1a481395b2ff3c4225987a596eb8f828cfcbb52766ad: Status 404 returned error can't find the container with id 373699f3ae8ca8ae3aad1a481395b2ff3c4225987a596eb8f828cfcbb52766ad
Jul 06 20:34:50.765132 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:50.765040    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19194284_20fc_4878_8adc_3e77e8cc918c.slice/crio-0997d20dbd38b9e1fc9e59e04b8207eac0588c51d4b9db47c4504c76052c8412 WatchSource:0}: Error finding container 0997d20dbd38b9e1fc9e59e04b8207eac0588c51d4b9db47c4504c76052c8412: Status 404 returned error can't find the container with id 0997d20dbd38b9e1fc9e59e04b8207eac0588c51d4b9db47c4504c76052c8412
Jul 06 20:34:50.767524 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:50.767495    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9353a6_63d4_4119_bcef_6cb39dfb6b2e.slice/crio-f32cd01397067f9e5484c3f095db47ec3d0526e750dc05ee862c66146943ca55 WatchSource:0}: Error finding container f32cd01397067f9e5484c3f095db47ec3d0526e750dc05ee862c66146943ca55: Status 404 returned error can't find the container with id f32cd01397067f9e5484c3f095db47ec3d0526e750dc05ee862c66146943ca55
Jul 06 20:34:50.768789 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:50.768768    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f4c0df_6488_42fa_b247_34d41d7284a7.slice/crio-1b56ee381d89efce59734f5ea9e4b5b74bfb2204eadf69a142a0ba8c44cc6cca WatchSource:0}: Error finding container 1b56ee381d89efce59734f5ea9e4b5b74bfb2204eadf69a142a0ba8c44cc6cca: Status 404 returned error can't find the container with id 1b56ee381d89efce59734f5ea9e4b5b74bfb2204eadf69a142a0ba8c44cc6cca
Jul 06 20:34:50.769465 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:50.769397    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f30a43_0971_4ca0_af13_9baba5202359.slice/crio-f2db36476240905cf05b9db0e8cf2a7cfc8dd14cca9e693c7552fc73a023e737 WatchSource:0}: Error finding container f2db36476240905cf05b9db0e8cf2a7cfc8dd14cca9e693c7552fc73a023e737: Status 404 returned error can't find the container with id f2db36476240905cf05b9db0e8cf2a7cfc8dd14cca9e693c7552fc73a023e737
Jul 06 20:34:50.770792 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:50.770764    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe996a8_9939_4be3_a9d2_a574dbc4ce24.slice/crio-dc0f668a11e3c668cc0f5be1a4ba5cb834a689941e2004b7aeac2246755ab170 WatchSource:0}: Error finding container dc0f668a11e3c668cc0f5be1a4ba5cb834a689941e2004b7aeac2246755ab170: Status 404 returned error can't find the container with id dc0f668a11e3c668cc0f5be1a4ba5cb834a689941e2004b7aeac2246755ab170
Jul 06 20:34:50.773513 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:34:50.772756    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d093092_ba3b_4059_ae54_cb0a6e2e4caf.slice/crio-0415f414d36892d390bde266293b64129cff71ae3f45e99153964ae40baac356 WatchSource:0}: Error finding container 0415f414d36892d390bde266293b64129cff71ae3f45e99153964ae40baac356: Status 404 returned error can't find the container with id 0415f414d36892d390bde266293b64129cff71ae3f45e99153964ae40baac356
Jul 06 20:34:50.974200 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:50.974153    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:50.974406 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:50.974218    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:50.974406 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:50.974332    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:34:50.974406 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:50.974377    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:34:50.974685 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:50.974421    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs podName:9866e3e7-f454-42b7-add8-3bcf2b9f797a nodeName:}" failed. No retries permitted until 2026-07-06 20:34:52.974398042 +0000 UTC m=+5.915701868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs") pod "network-metrics-daemon-6vs4n" (UID: "9866e3e7-f454-42b7-add8-3bcf2b9f797a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:34:50.974685 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:50.974442    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret podName:1cd6de51-6683-4b89-b584-0cb22a3b8833 nodeName:}" failed. No retries permitted until 2026-07-06 20:34:52.974431785 +0000 UTC m=+5.915735616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret") pod "global-pull-secret-syncer-dnd82" (UID: "1cd6de51-6683-4b89-b584-0cb22a3b8833") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:34:51.175505 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.175451    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248l8\" (UniqueName: \"kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8\") pod \"network-check-target-kkslg\" (UID: \"7510d9cd-07f1-4c52-ad19-3a31394c9300\") " pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:51.175690 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:51.175610    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:34:51.175690 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:51.175630    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:34:51.175690 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:51.175642    2550 projected.go:196] Error preparing data for projected volume kube-api-access-248l8 for pod openshift-network-diagnostics/network-check-target-kkslg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:34:51.175871 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:51.175707    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8 podName:7510d9cd-07f1-4c52-ad19-3a31394c9300 nodeName:}" failed. No retries permitted until 2026-07-06 20:34:53.175685822 +0000 UTC m=+6.116989651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-248l8" (UniqueName: "kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8") pod "network-check-target-kkslg" (UID: "7510d9cd-07f1-4c52-ad19-3a31394c9300") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:34:51.414239 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.414170    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:29:49 +0000 UTC" deadline="2028-03-28 17:37:07.102373226 +0000 UTC"
Jul 06 20:34:51.414239 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.414207    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15141h2m15.688171133s"
Jul 06 20:34:51.471759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.470761    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:51.471759 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:51.470912    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:34:51.471759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.471410    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:51.471759 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:51.471523    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:34:51.471759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.471608    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:51.471759 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:51.471699    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:34:51.494403 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.493776    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" event={"ID":"3e825bc10f7326db6e76e5c99478ee31","Type":"ContainerStarted","Data":"c5fec70aa9e5a1566f61beca3f46f32d741214544ab45104a6de9773acb87f80"}
Jul 06 20:34:51.502792 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.502725    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mqszb" event={"ID":"6fe996a8-9939-4be3-a9d2-a574dbc4ce24","Type":"ContainerStarted","Data":"dc0f668a11e3c668cc0f5be1a4ba5cb834a689941e2004b7aeac2246755ab170"}
Jul 06 20:34:51.507300 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.507272    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lgsmp" event={"ID":"77f30a43-0971-4ca0-af13-9baba5202359","Type":"ContainerStarted","Data":"f2db36476240905cf05b9db0e8cf2a7cfc8dd14cca9e693c7552fc73a023e737"}
Jul 06 20:34:51.512519 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.512491    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn5nv" event={"ID":"5d093092-ba3b-4059-ae54-cb0a6e2e4caf","Type":"ContainerStarted","Data":"0415f414d36892d390bde266293b64129cff71ae3f45e99153964ae40baac356"}
Jul 06 20:34:51.518960 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.518931    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lqxtl" event={"ID":"e4f4c0df-6488-42fa-b247-34d41d7284a7","Type":"ContainerStarted","Data":"1b56ee381d89efce59734f5ea9e4b5b74bfb2204eadf69a142a0ba8c44cc6cca"}
Jul 06 20:34:51.531685 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.531547    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn" event={"ID":"19194284-20fc-4878-8adc-3e77e8cc918c","Type":"ContainerStarted","Data":"0997d20dbd38b9e1fc9e59e04b8207eac0588c51d4b9db47c4504c76052c8412"}
Jul 06 20:34:51.534686 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.534659    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fp74t" event={"ID":"5eef1e47-1131-4820-b054-5ad627772e0f","Type":"ContainerStarted","Data":"373699f3ae8ca8ae3aad1a481395b2ff3c4225987a596eb8f828cfcbb52766ad"}
Jul 06 20:34:51.555926 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.555091    2550 generic.go:356] "Generic (PLEG): container finished" podID="45a32833bf4d608b8ecac078de2241ac" containerID="63bdfdab198284762c8f74e8f1b2c95a9bcc5e0ca5688866a0f6fa32ac2bb881" exitCode=0
Jul 06 20:34:51.555926 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.555175    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" event={"ID":"45a32833bf4d608b8ecac078de2241ac","Type":"ContainerDied","Data":"63bdfdab198284762c8f74e8f1b2c95a9bcc5e0ca5688866a0f6fa32ac2bb881"}
Jul 06 20:34:51.561508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.561414    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m7rjb" event={"ID":"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e","Type":"ContainerStarted","Data":"f32cd01397067f9e5484c3f095db47ec3d0526e750dc05ee862c66146943ca55"}
Jul 06 20:34:51.565382 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.565301    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zbdwk" event={"ID":"8f090afa-8dff-4167-b753-523344760615","Type":"ContainerStarted","Data":"0fa6cfa6c2b1df5c953bca615569910362acf83286c1780c970985f3eb071a49"}
Jul 06 20:34:51.577508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:51.577356    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" podStartSLOduration=3.577337254 podStartE2EDuration="3.577337254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:34:51.520733125 +0000 UTC m=+4.462036978" watchObservedRunningTime="2026-07-06 20:34:51.577337254 +0000 UTC m=+4.518641105"
Jul 06 20:34:52.595721 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:52.595611    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" event={"ID":"45a32833bf4d608b8ecac078de2241ac","Type":"ContainerStarted","Data":"ed233ef6c0bed86747195b317eefa2d7e036451458fa0648e301bb4c19e4f4f5"}
Jul 06 20:34:52.997060 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:52.996972    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:52.997253 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:52.997084    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:52.997253 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:52.997226    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:34:52.997374 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:52.997295    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs podName:9866e3e7-f454-42b7-add8-3bcf2b9f797a nodeName:}" failed. No retries permitted until 2026-07-06 20:34:56.997273396 +0000 UTC m=+9.938577227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs") pod "network-metrics-daemon-6vs4n" (UID: "9866e3e7-f454-42b7-add8-3bcf2b9f797a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:34:52.997720 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:52.997698    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:34:52.997831 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:52.997758    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret podName:1cd6de51-6683-4b89-b584-0cb22a3b8833 nodeName:}" failed. No retries permitted until 2026-07-06 20:34:56.997742419 +0000 UTC m=+9.939046250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret") pod "global-pull-secret-syncer-dnd82" (UID: "1cd6de51-6683-4b89-b584-0cb22a3b8833") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:34:53.198496 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:53.198458    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248l8\" (UniqueName: \"kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8\") pod \"network-check-target-kkslg\" (UID: \"7510d9cd-07f1-4c52-ad19-3a31394c9300\") " pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:53.198732 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:53.198691    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:34:53.198732 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:53.198715    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:34:53.198732 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:53.198729    2550 projected.go:196] Error preparing data for projected volume kube-api-access-248l8 for pod openshift-network-diagnostics/network-check-target-kkslg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:34:53.198960 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:53.198791    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8 podName:7510d9cd-07f1-4c52-ad19-3a31394c9300 nodeName:}" failed. No retries permitted until 2026-07-06 20:34:57.1987729 +0000 UTC m=+10.140076748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-248l8" (UniqueName: "kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8") pod "network-check-target-kkslg" (UID: "7510d9cd-07f1-4c52-ad19-3a31394c9300") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:34:53.468793 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:53.468199    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:53.468793 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:53.468208    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:53.468793 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:53.468355    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:53.468793 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:53.468348    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:34:53.468793 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:53.468458    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:34:53.468793 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:53.468611    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:34:55.468588 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:55.468041    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:55.468588 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:55.468178    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:34:55.468588 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:55.468199    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:55.468588 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:55.468318    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:34:55.468588 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:55.468366    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:55.468588 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:55.468443    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:34:57.031546 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:57.031448    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:57.031546 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:57.031510    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:57.032072 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.031630    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:34:57.032072 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.031637    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:34:57.032072 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.031702    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret podName:1cd6de51-6683-4b89-b584-0cb22a3b8833 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:05.031680648 +0000 UTC m=+17.972984476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret") pod "global-pull-secret-syncer-dnd82" (UID: "1cd6de51-6683-4b89-b584-0cb22a3b8833") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:34:57.032072 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.031724    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs podName:9866e3e7-f454-42b7-add8-3bcf2b9f797a nodeName:}" failed. No retries permitted until 2026-07-06 20:35:05.031712976 +0000 UTC m=+17.973016804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs") pod "network-metrics-daemon-6vs4n" (UID: "9866e3e7-f454-42b7-add8-3bcf2b9f797a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:34:57.233680 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:57.233471    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248l8\" (UniqueName: \"kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8\") pod \"network-check-target-kkslg\" (UID: \"7510d9cd-07f1-4c52-ad19-3a31394c9300\") " pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:57.233680 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.233640    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:34:57.233680 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.233659    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:34:57.233680 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.233671    2550 projected.go:196] Error preparing data for projected volume kube-api-access-248l8 for pod openshift-network-diagnostics/network-check-target-kkslg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:34:57.234013 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.233733    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8 podName:7510d9cd-07f1-4c52-ad19-3a31394c9300 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:05.233715369 +0000 UTC m=+18.175019195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-248l8" (UniqueName: "kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8") pod "network-check-target-kkslg" (UID: "7510d9cd-07f1-4c52-ad19-3a31394c9300") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:34:57.469573 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:57.469542    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:57.469766 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.469681    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:34:57.470046 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:57.470026    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:57.470144 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.470123    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:34:57.470193 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:57.470177    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:57.470272 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:57.470255    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:34:59.468622 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:59.468127    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:34:59.468622 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:59.468134    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:34:59.468622 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:59.468270    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:34:59.468622 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:34:59.468144    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:34:59.468622 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:59.468356    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:34:59.468622 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:34:59.468402    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:01.468046 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:01.467997    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:01.468046 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:01.468038    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:01.468528 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:01.467997    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:01.468528 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:01.468128    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:01.468528 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:01.468196    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:01.468528 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:01.468280    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:03.468675 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:03.468646    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:03.469129 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:03.468646    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:03.469129 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:03.468786    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:03.469129 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:03.468646    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:03.469129 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:03.468876    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:03.469129 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:03.468961    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:05.091312 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:05.091284    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:05.091760 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:05.091330    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:05.091760 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.091438    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:05.091760 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.091445    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:05.091760 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.091492    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret podName:1cd6de51-6683-4b89-b584-0cb22a3b8833 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:21.091474922 +0000 UTC m=+34.032778768 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret") pod "global-pull-secret-syncer-dnd82" (UID: "1cd6de51-6683-4b89-b584-0cb22a3b8833") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:05.091760 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.091512    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs podName:9866e3e7-f454-42b7-add8-3bcf2b9f797a nodeName:}" failed. No retries permitted until 2026-07-06 20:35:21.091501714 +0000 UTC m=+34.032805542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs") pod "network-metrics-daemon-6vs4n" (UID: "9866e3e7-f454-42b7-add8-3bcf2b9f797a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:05.293248 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:05.293219    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248l8\" (UniqueName: \"kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8\") pod \"network-check-target-kkslg\" (UID: \"7510d9cd-07f1-4c52-ad19-3a31394c9300\") " pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:05.293395 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.293341    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:05.293395 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.293354    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:05.293395 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.293363    2550 projected.go:196] Error preparing data for projected volume kube-api-access-248l8 for pod openshift-network-diagnostics/network-check-target-kkslg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:05.293490 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.293410    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8 podName:7510d9cd-07f1-4c52-ad19-3a31394c9300 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:21.293393055 +0000 UTC m=+34.234696880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-248l8" (UniqueName: "kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8") pod "network-check-target-kkslg" (UID: "7510d9cd-07f1-4c52-ad19-3a31394c9300") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:05.468203 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:05.468128    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:05.468203 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:05.468149    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:05.468413 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:05.468242    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:05.468413 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.468259    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:05.468413 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.468339    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:05.468413 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:05.468400    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:07.469372 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:07.469335    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:07.469827 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:07.469447    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:07.469827 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:07.469508    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:07.469827 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:07.469640    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:07.469827 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:07.469675    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:07.469827 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:07.469722    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:09.468082 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:09.468055    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:09.468082 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:09.468072    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:09.468505 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:09.468163    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:09.468505 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:09.468169    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:09.468505 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:09.468277    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:09.468505 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:09.468452    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:09.642472 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:09.642254    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m7rjb" event={"ID":"ea9353a6-63d4-4119-bcef-6cb39dfb6b2e","Type":"ContainerStarted","Data":"1dd597417423cbabfd7ae41c63965911f74d80f548ad2fac46b1ea8f41bf7fe5"}
Jul 06 20:35:09.647593 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:09.647533    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lgsmp" event={"ID":"77f30a43-0971-4ca0-af13-9baba5202359","Type":"ContainerStarted","Data":"854a82175f670cffe289cd36d53d7a6500c75ad3babc281b83bca1e784c59d22"}
Jul 06 20:35:09.664341 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:09.664280    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m7rjb" podStartSLOduration=2.9761590399999998 podStartE2EDuration="21.6642583s" totalImagesPullingTime="18.68809926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:09.663138121 +0000 UTC m=+22.604442010" watchObservedRunningTime="2026-07-06 20:35:09.6642583 +0000 UTC m=+22.605562148"
Jul 06 20:35:09.664465 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:09.664434    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" podStartSLOduration=21.664427181 podStartE2EDuration="21.664427181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:34:52.627061804 +0000 UTC m=+5.568365657" watchObservedRunningTime="2026-07-06 20:35:09.664427181 +0000 UTC m=+22.605731029"
Jul 06 20:35:10.654801 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.654611    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mqszb" event={"ID":"6fe996a8-9939-4be3-a9d2-a574dbc4ce24","Type":"ContainerStarted","Data":"a8d1e32ba3b6ff1c50771a9c83c0efb624d0f0120818c22e21a4caec7c8f94be"}
Jul 06 20:35:10.656128 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.656105    2550 generic.go:356] "Generic (PLEG): container finished" podID="5d093092-ba3b-4059-ae54-cb0a6e2e4caf" containerID="5f988859d4c04a4da8bb8a8f6e7e9fa69477ae9ab32e8c24d8ac2e6271cf70b6" exitCode=0
Jul 06 20:35:10.656235 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.656172    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn5nv" event={"ID":"5d093092-ba3b-4059-ae54-cb0a6e2e4caf","Type":"ContainerDied","Data":"5f988859d4c04a4da8bb8a8f6e7e9fa69477ae9ab32e8c24d8ac2e6271cf70b6"}
Jul 06 20:35:10.659669 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.659641    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lqxtl" event={"ID":"e4f4c0df-6488-42fa-b247-34d41d7284a7","Type":"ContainerStarted","Data":"2fb5812c8a6ce292f398e90c8d064160c9d2fbaf6994c6b9c01a90ef55d0ce62"}
Jul 06 20:35:10.662478 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.662458    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn" event={"ID":"19194284-20fc-4878-8adc-3e77e8cc918c","Type":"ContainerStarted","Data":"aaaf5a6c5388c4f563308debe080baddcd1424c79b547b8eff04072288d14262"}
Jul 06 20:35:10.662572 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.662509    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn" event={"ID":"19194284-20fc-4878-8adc-3e77e8cc918c","Type":"ContainerStarted","Data":"8ba87e48cb1039f24edc3e9a701e530ca43e10b79d1a39640424150bdf8f9cc1"}
Jul 06 20:35:10.662572 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.662545    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn" event={"ID":"19194284-20fc-4878-8adc-3e77e8cc918c","Type":"ContainerStarted","Data":"1d2a1d48636e93e7068f57f51ce2e1a5e7977c964e8f51cfad1aae4ad7f26852"}
Jul 06 20:35:10.662694 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.662581    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn" event={"ID":"19194284-20fc-4878-8adc-3e77e8cc918c","Type":"ContainerStarted","Data":"ff297f3d3ff44381001b635aac599aec245dcda858be93494e7003ac9b67100d"}
Jul 06 20:35:10.662694 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.662622    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn" event={"ID":"19194284-20fc-4878-8adc-3e77e8cc918c","Type":"ContainerStarted","Data":"5fd1a3514b55696a2bc73accda99815a7c65036bfc44601bdf5434b2d759f774"}
Jul 06 20:35:10.662694 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.662668    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn" event={"ID":"19194284-20fc-4878-8adc-3e77e8cc918c","Type":"ContainerStarted","Data":"4090301baf62ce07bf90997ab8048f10df0d72db07227644f15254660914278b"}
Jul 06 20:35:10.663760 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.663742    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fp74t" event={"ID":"5eef1e47-1131-4820-b054-5ad627772e0f","Type":"ContainerStarted","Data":"c9441f27d2907567085e7613d9d27f0a7c382a65f87a6b064082b0ab08010ecc"}
Jul 06 20:35:10.665016 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.664991    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7" event={"ID":"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f","Type":"ContainerStarted","Data":"6c59bba4390c859448bef72732952ab371e2c8dc4c7eb0c9efbbe57e08812e18"}
Jul 06 20:35:10.666231 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.666212    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zbdwk" event={"ID":"8f090afa-8dff-4167-b753-523344760615","Type":"ContainerStarted","Data":"bdf7fdf76f6826e1bffa9df4c32b006d4263311b1b67b81f55ad2e67f828e74c"}
Jul 06 20:35:10.678032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.677997    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mqszb" podStartSLOduration=3.9904987910000003 podStartE2EDuration="22.677986207s" totalImagesPullingTime="18.687487416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:10.677875073 +0000 UTC m=+23.619178920" watchObservedRunningTime="2026-07-06 20:35:10.677986207 +0000 UTC m=+23.619290053"
Jul 06 20:35:10.678115 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.678059    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lgsmp" podStartSLOduration=3.937066367 podStartE2EDuration="22.678053891s" totalImagesPullingTime="18.740987524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:09.693385745 +0000 UTC m=+22.634689592" watchObservedRunningTime="2026-07-06 20:35:10.678053891 +0000 UTC m=+23.619357738"
Jul 06 20:35:10.730384 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.730349    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lqxtl" podStartSLOduration=4.040747707 podStartE2EDuration="22.73033637s" totalImagesPullingTime="18.689588663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:10.730144059 +0000 UTC m=+23.671447905" watchObservedRunningTime="2026-07-06 20:35:10.73033637 +0000 UTC m=+23.671640218"
Jul 06 20:35:10.748739 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.748666    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zbdwk" podStartSLOduration=4.051699243 podStartE2EDuration="22.748655144s" totalImagesPullingTime="18.696955901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:10.748178679 +0000 UTC m=+23.689482537" watchObservedRunningTime="2026-07-06 20:35:10.748655144 +0000 UTC m=+23.689958991"
Jul 06 20:35:10.769705 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:10.769641    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fp74t" podStartSLOduration=4.117646717 podStartE2EDuration="22.769628909s" totalImagesPullingTime="18.651982192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:10.769467744 +0000 UTC m=+23.710771592" watchObservedRunningTime="2026-07-06 20:35:10.769628909 +0000 UTC m=+23.710932750"
Jul 06 20:35:11.066029 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:11.066007    2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 20:35:11.415545 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:11.415420    2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:35:11.066025031Z","UUID":"b14519f0-bcc8-4049-b1a6-40b5de7c9d7c","Handler":null,"Name":"","Endpoint":""}
Jul 06 20:35:11.416699 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:11.416681    2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 20:35:11.416824 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:11.416716    2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 20:35:11.468346 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:11.468315    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:11.468482 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:11.468432    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:11.468482 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:11.468441    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:11.468601 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:11.468545    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:11.468601 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:11.468575    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:11.468670 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:11.468657    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:11.671423 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:11.671344    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7" event={"ID":"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f","Type":"ContainerStarted","Data":"907300b893424142394d8225c88d26636129d87cf37ac60e76076abb71ee8577"}
Jul 06 20:35:12.677865 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:12.677824    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn" event={"ID":"19194284-20fc-4878-8adc-3e77e8cc918c","Type":"ContainerStarted","Data":"bdf8f79ded5f683ff6be0da19df9409b6fcf2fe6286ef2b5920e130069960004"}
Jul 06 20:35:12.680263 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:12.680229    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7" event={"ID":"7ef4bba0-0c6e-499f-a5d2-7eae1968e21f","Type":"ContainerStarted","Data":"4a7d194aabc2c0d18cf67f3041018e0ec0235ac66251172c3937e13636a9708f"}
Jul 06 20:35:12.702626 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:12.702560    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl2j7" podStartSLOduration=11.101312469 podStartE2EDuration="24.702541058s" totalImagesPullingTime="13.601228589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:12.700786136 +0000 UTC m=+25.642089983" watchObservedRunningTime="2026-07-06 20:35:12.702541058 +0000 UTC m=+25.643844901"
Jul 06 20:35:13.246764 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:13.246473    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:35:13.247176 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:13.247152    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:35:13.468614 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:13.468574    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:13.468771 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:13.468574    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:13.468771 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:13.468688    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:13.468771 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:13.468695    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:13.468948 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:13.468775    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:13.468948 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:13.468863    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:13.682399 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:13.682363    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:35:13.682935 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:13.682918    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zbdwk"
Jul 06 20:35:15.468664 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:15.468490    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:15.469332 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:15.468521    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:15.469332 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:15.468740    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:15.469332 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:15.468544    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:15.469332 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:15.468833    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:15.469332 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:15.468933    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:15.690742 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:15.690653    2550 generic.go:356] "Generic (PLEG): container finished" podID="5d093092-ba3b-4059-ae54-cb0a6e2e4caf" containerID="ef5cd32cfe8f577ee42dbab46de6eff8caee5fffa9bcc2ad9f8bfca635d18b4b" exitCode=0
Jul 06 20:35:15.690914 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:15.690741    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn5nv" event={"ID":"5d093092-ba3b-4059-ae54-cb0a6e2e4caf","Type":"ContainerDied","Data":"ef5cd32cfe8f577ee42dbab46de6eff8caee5fffa9bcc2ad9f8bfca635d18b4b"}
Jul 06 20:35:15.694662 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:15.694640    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn" event={"ID":"19194284-20fc-4878-8adc-3e77e8cc918c","Type":"ContainerStarted","Data":"75f97159d4e8a31a2bc530063f86c3e5e4fb389509f2c139e6ced805ef454d98"}
Jul 06 20:35:15.759930 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:15.759880    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn" podStartSLOduration=8.977007197 podStartE2EDuration="27.75986571s" totalImagesPullingTime="18.782858513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:15.75828138 +0000 UTC m=+28.699585227" watchObservedRunningTime="2026-07-06 20:35:15.75986571 +0000 UTC m=+28.701169553"
Jul 06 20:35:16.698846 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:16.698801    2550 generic.go:356] "Generic (PLEG): container finished" podID="5d093092-ba3b-4059-ae54-cb0a6e2e4caf" containerID="00e467bfc2d5c4322614e151da82f5f889b116d6bc7020803f8eb97e0214e513" exitCode=0
Jul 06 20:35:16.698846 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:16.698841    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn5nv" event={"ID":"5d093092-ba3b-4059-ae54-cb0a6e2e4caf","Type":"ContainerDied","Data":"00e467bfc2d5c4322614e151da82f5f889b116d6bc7020803f8eb97e0214e513"}
Jul 06 20:35:16.699451 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:16.699412    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:35:16.699550 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:16.699497    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:35:16.699550 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:16.699547    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:35:16.713585 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:16.713563    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:35:16.713829 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:16.713740    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:35:17.466943 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:17.466908    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kkslg"]
Jul 06 20:35:17.467085 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:17.467034    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:17.467126 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:17.467109    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:17.468921 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:17.468897    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:17.468990 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:17.468972    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:17.469041 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:17.469014    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:17.469072 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:17.469058    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:17.473495 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:17.473464    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dnd82"]
Jul 06 20:35:17.497402 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:17.497376    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6vs4n"]
Jul 06 20:35:17.701879 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:17.701565    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:17.701879 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:17.701700    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:17.701879 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:17.701714    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:17.702346 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:17.702124    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:18.706501 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:18.706425    2550 generic.go:356] "Generic (PLEG): container finished" podID="5d093092-ba3b-4059-ae54-cb0a6e2e4caf" containerID="efe4abad940b73912906905f8b195e21e46b57ae0be655751433d55ff9e37420" exitCode=0
Jul 06 20:35:18.706907 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:18.706498    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn5nv" event={"ID":"5d093092-ba3b-4059-ae54-cb0a6e2e4caf","Type":"ContainerDied","Data":"efe4abad940b73912906905f8b195e21e46b57ae0be655751433d55ff9e37420"}
Jul 06 20:35:19.468379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:19.468293    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:19.468379 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:19.468293    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:19.468564 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:19.468432    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:19.468564 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:19.468455    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:19.468633 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:19.468604    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:19.468756 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:19.468729    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:21.114362 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:21.114182    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:21.114801 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:21.114373    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:21.114801 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.114323    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:21.114801 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.114456    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:21.114801 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.114501    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs podName:9866e3e7-f454-42b7-add8-3bcf2b9f797a nodeName:}" failed. No retries permitted until 2026-07-06 20:35:53.114477929 +0000 UTC m=+66.055781769 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs") pod "network-metrics-daemon-6vs4n" (UID: "9866e3e7-f454-42b7-add8-3bcf2b9f797a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:21.114801 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.114521    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret podName:1cd6de51-6683-4b89-b584-0cb22a3b8833 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:53.114511795 +0000 UTC m=+66.055815625 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret") pod "global-pull-secret-syncer-dnd82" (UID: "1cd6de51-6683-4b89-b584-0cb22a3b8833") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:21.315926 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:21.315891    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248l8\" (UniqueName: \"kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8\") pod \"network-check-target-kkslg\" (UID: \"7510d9cd-07f1-4c52-ad19-3a31394c9300\") " pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:21.316111 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.316031    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:21.316111 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.316048    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:21.316111 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.316061    2550 projected.go:196] Error preparing data for projected volume kube-api-access-248l8 for pod openshift-network-diagnostics/network-check-target-kkslg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:21.316260 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.316120    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8 podName:7510d9cd-07f1-4c52-ad19-3a31394c9300 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:53.316101754 +0000 UTC m=+66.257405586 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-248l8" (UniqueName: "kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8") pod "network-check-target-kkslg" (UID: "7510d9cd-07f1-4c52-ad19-3a31394c9300") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:21.468177 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:21.468094    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:21.468315 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:21.468106    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:21.468315 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.468230    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vs4n" podUID="9866e3e7-f454-42b7-add8-3bcf2b9f797a"
Jul 06 20:35:21.468315 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:21.468271    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:21.468470 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.468393    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kkslg" podUID="7510d9cd-07f1-4c52-ad19-3a31394c9300"
Jul 06 20:35:21.468520 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:21.468479    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnd82" podUID="1cd6de51-6683-4b89-b584-0cb22a3b8833"
Jul 06 20:35:22.457420 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.457382    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeReady"
Jul 06 20:35:22.457922 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.457520    2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 20:35:22.495048 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.495015    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7558884f8-xtl65"]
Jul 06 20:35:22.523091 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.523044    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-57cbl"]
Jul 06 20:35:22.523371 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.523346    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.526849 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.526637    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 20:35:22.526849 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.526701    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nc576\""
Jul 06 20:35:22.526849 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.526718    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 20:35:22.527023 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.526997    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 20:35:22.533780 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.533756    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 20:35:22.536939 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.536919    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nb4m7"]
Jul 06 20:35:22.537072 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.537051    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-57cbl"
Jul 06 20:35:22.540057 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.540021    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 20:35:22.540413 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.540394    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r4sgm\""
Jul 06 20:35:22.540631 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.540615    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 20:35:22.540790 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.540637    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 20:35:22.556689 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.556671    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7558884f8-xtl65"]
Jul 06 20:35:22.556781 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.556703    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-57cbl"]
Jul 06 20:35:22.556781 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.556725    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nb4m7"]
Jul 06 20:35:22.556883 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.556801    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.559351 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.559333    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 20:35:22.559458 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.559382    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 20:35:22.559872 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.559854    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dqcct\""
Jul 06 20:35:22.624106 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.624074    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/354a41da-1c7d-4c43-a17c-bcea3ede2a96-ca-trust-extracted\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.624230 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.624113    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-installation-pull-secrets\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.624230 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.624174    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgf5\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-kube-api-access-fqgf5\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.624230 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.624203    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.624368 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.624232    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-certificates\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.624368 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.624256    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-trusted-ca\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.624368 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.624280    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-bound-sa-token\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.624368 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.624320    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-image-registry-private-configuration\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.725231 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725134    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-certificates\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.725231 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725167    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-trusted-ca\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.725231 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725195    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-bound-sa-token\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.725507 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725241    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.725507 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725267    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6624a7-c191-44b2-806f-c1a04a48402e-tmp-dir\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.725507 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725298    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnw5v\" (UniqueName: \"kubernetes.io/projected/3c6624a7-c191-44b2-806f-c1a04a48402e-kube-api-access-mnw5v\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.725507 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725331    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-image-registry-private-configuration\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.725507 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725363    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6624a7-c191-44b2-806f-c1a04a48402e-config-volume\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.725507 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725391    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh875\" (UniqueName: \"kubernetes.io/projected/ec945c9f-5716-49f2-8dd7-0347fe6275dd-kube-api-access-qh875\") pod \"ingress-canary-57cbl\" (UID: \"ec945c9f-5716-49f2-8dd7-0347fe6275dd\") " pod="openshift-ingress-canary/ingress-canary-57cbl"
Jul 06 20:35:22.725788 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725562    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec945c9f-5716-49f2-8dd7-0347fe6275dd-cert\") pod \"ingress-canary-57cbl\" (UID: \"ec945c9f-5716-49f2-8dd7-0347fe6275dd\") " pod="openshift-ingress-canary/ingress-canary-57cbl"
Jul 06 20:35:22.725788 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725613    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/354a41da-1c7d-4c43-a17c-bcea3ede2a96-ca-trust-extracted\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.725788 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725639    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-installation-pull-secrets\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.725961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725791    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgf5\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-kube-api-access-fqgf5\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.725961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725853    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.725961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725897    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-certificates\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.725961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.725957    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/354a41da-1c7d-4c43-a17c-bcea3ede2a96-ca-trust-extracted\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.726145 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:22.725981    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:35:22.726145 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:22.725997    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7558884f8-xtl65: secret "image-registry-tls" not found
Jul 06 20:35:22.726145 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:22.726066    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls podName:354a41da-1c7d-4c43-a17c-bcea3ede2a96 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:23.226047469 +0000 UTC m=+36.167351293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls") pod "image-registry-7558884f8-xtl65" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96") : secret "image-registry-tls" not found
Jul 06 20:35:22.726286 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.726148    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-trusted-ca\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.730654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.730611    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-image-registry-private-configuration\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.730654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.730623    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-installation-pull-secrets\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.734992 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.734947    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-bound-sa-token\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.735329 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.735303    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgf5\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-kube-api-access-fqgf5\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:22.826509 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.826467    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.826509 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.826520    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6624a7-c191-44b2-806f-c1a04a48402e-tmp-dir\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.826759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.826549    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnw5v\" (UniqueName: \"kubernetes.io/projected/3c6624a7-c191-44b2-806f-c1a04a48402e-kube-api-access-mnw5v\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.826759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.826582    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6624a7-c191-44b2-806f-c1a04a48402e-config-volume\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.826759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.826605    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh875\" (UniqueName: \"kubernetes.io/projected/ec945c9f-5716-49f2-8dd7-0347fe6275dd-kube-api-access-qh875\") pod \"ingress-canary-57cbl\" (UID: \"ec945c9f-5716-49f2-8dd7-0347fe6275dd\") " pod="openshift-ingress-canary/ingress-canary-57cbl"
Jul 06 20:35:22.826759 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:22.826642    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:22.826759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.826654    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec945c9f-5716-49f2-8dd7-0347fe6275dd-cert\") pod \"ingress-canary-57cbl\" (UID: \"ec945c9f-5716-49f2-8dd7-0347fe6275dd\") " pod="openshift-ingress-canary/ingress-canary-57cbl"
Jul 06 20:35:22.826759 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:22.826730    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls podName:3c6624a7-c191-44b2-806f-c1a04a48402e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:23.326706491 +0000 UTC m=+36.268010336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls") pod "dns-default-nb4m7" (UID: "3c6624a7-c191-44b2-806f-c1a04a48402e") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:22.827067 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.826880    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6624a7-c191-44b2-806f-c1a04a48402e-tmp-dir\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.827387 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.827358    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6624a7-c191-44b2-806f-c1a04a48402e-config-volume\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.829317 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.829295    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec945c9f-5716-49f2-8dd7-0347fe6275dd-cert\") pod \"ingress-canary-57cbl\" (UID: \"ec945c9f-5716-49f2-8dd7-0347fe6275dd\") " pod="openshift-ingress-canary/ingress-canary-57cbl"
Jul 06 20:35:22.835922 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.835894    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnw5v\" (UniqueName: \"kubernetes.io/projected/3c6624a7-c191-44b2-806f-c1a04a48402e-kube-api-access-mnw5v\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:22.836428 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.836402    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh875\" (UniqueName: \"kubernetes.io/projected/ec945c9f-5716-49f2-8dd7-0347fe6275dd-kube-api-access-qh875\") pod \"ingress-canary-57cbl\" (UID: \"ec945c9f-5716-49f2-8dd7-0347fe6275dd\") " pod="openshift-ingress-canary/ingress-canary-57cbl"
Jul 06 20:35:22.847087 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:22.847067    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-57cbl"
Jul 06 20:35:23.230277 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.230247    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:23.230459 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:23.230418    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:35:23.230459 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:23.230437    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7558884f8-xtl65: secret "image-registry-tls" not found
Jul 06 20:35:23.230556 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:23.230496    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls podName:354a41da-1c7d-4c43-a17c-bcea3ede2a96 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:24.230478207 +0000 UTC m=+37.171782034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls") pod "image-registry-7558884f8-xtl65" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96") : secret "image-registry-tls" not found
Jul 06 20:35:23.331265 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.331237    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:23.331417 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:23.331398    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:23.331478 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:23.331455    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls podName:3c6624a7-c191-44b2-806f-c1a04a48402e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:24.331439848 +0000 UTC m=+37.272743673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls") pod "dns-default-nb4m7" (UID: "3c6624a7-c191-44b2-806f-c1a04a48402e") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:23.468034 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.467997    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:23.468654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.467998    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:23.468654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.468128    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:23.471110 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.471085    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:35:23.471494 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.471236    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl572\""
Jul 06 20:35:23.471494 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.471297    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:35:23.471494 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.471315    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:35:23.471494 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.471325    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:35:23.471494 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.471336    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6bf8b\""
Jul 06 20:35:23.761188 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:23.761057    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-57cbl"]
Jul 06 20:35:23.829902 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:35:23.829872    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec945c9f_5716_49f2_8dd7_0347fe6275dd.slice/crio-124b9aee5b9ab3114b2d68cabd1312fb48395459c997a6442de0a684d0c36df9 WatchSource:0}: Error finding container 124b9aee5b9ab3114b2d68cabd1312fb48395459c997a6442de0a684d0c36df9: Status 404 returned error can't find the container with id 124b9aee5b9ab3114b2d68cabd1312fb48395459c997a6442de0a684d0c36df9
Jul 06 20:35:24.254233 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:24.254058    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:24.254357 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:24.254199    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:35:24.254357 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:24.254281    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7558884f8-xtl65: secret "image-registry-tls" not found
Jul 06 20:35:24.254357 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:24.254342    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls podName:354a41da-1c7d-4c43-a17c-bcea3ede2a96 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:26.254327339 +0000 UTC m=+39.195631176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls") pod "image-registry-7558884f8-xtl65" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96") : secret "image-registry-tls" not found
Jul 06 20:35:24.354495 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:24.354470    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:24.354598 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:24.354562    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:24.354631 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:24.354609    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls podName:3c6624a7-c191-44b2-806f-c1a04a48402e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:26.354596084 +0000 UTC m=+39.295899909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls") pod "dns-default-nb4m7" (UID: "3c6624a7-c191-44b2-806f-c1a04a48402e") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:24.725926 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:24.725799    2550 generic.go:356] "Generic (PLEG): container finished" podID="5d093092-ba3b-4059-ae54-cb0a6e2e4caf" containerID="322c97c91dbfb3612c91719b02170ce7608b66d26c5c405d555a3143e234461e" exitCode=0
Jul 06 20:35:24.726651 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:24.725926    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn5nv" event={"ID":"5d093092-ba3b-4059-ae54-cb0a6e2e4caf","Type":"ContainerDied","Data":"322c97c91dbfb3612c91719b02170ce7608b66d26c5c405d555a3143e234461e"}
Jul 06 20:35:24.727306 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:24.727276    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-57cbl" event={"ID":"ec945c9f-5716-49f2-8dd7-0347fe6275dd","Type":"ContainerStarted","Data":"124b9aee5b9ab3114b2d68cabd1312fb48395459c997a6442de0a684d0c36df9"}
Jul 06 20:35:25.733482 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:25.733446    2550 generic.go:356] "Generic (PLEG): container finished" podID="5d093092-ba3b-4059-ae54-cb0a6e2e4caf" containerID="bba8e92b77a49d3f2e46e754de08df7cb11865106fbe476f64844d329a3fdc90" exitCode=0
Jul 06 20:35:25.733887 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:25.733527    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn5nv" event={"ID":"5d093092-ba3b-4059-ae54-cb0a6e2e4caf","Type":"ContainerDied","Data":"bba8e92b77a49d3f2e46e754de08df7cb11865106fbe476f64844d329a3fdc90"}
Jul 06 20:35:26.268666 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:26.268641    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:26.268803 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:26.268756    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:35:26.268803 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:26.268766    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7558884f8-xtl65: secret "image-registry-tls" not found
Jul 06 20:35:26.268893 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:26.268827    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls podName:354a41da-1c7d-4c43-a17c-bcea3ede2a96 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:30.268798613 +0000 UTC m=+43.210102439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls") pod "image-registry-7558884f8-xtl65" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96") : secret "image-registry-tls" not found
Jul 06 20:35:26.369323 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:26.369300    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:26.369444 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:26.369433    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:26.369486 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:26.369480    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls podName:3c6624a7-c191-44b2-806f-c1a04a48402e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:30.369466927 +0000 UTC m=+43.310770752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls") pod "dns-default-nb4m7" (UID: "3c6624a7-c191-44b2-806f-c1a04a48402e") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:26.738586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:26.738558    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn5nv" event={"ID":"5d093092-ba3b-4059-ae54-cb0a6e2e4caf","Type":"ContainerStarted","Data":"bd4615211b75c6a42e799edd8471caeaf650af1c22f56a28a5c9a4d8d17d66d9"}
Jul 06 20:35:26.776294 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:26.776250    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jn5nv" podStartSLOduration=8.613889621 podStartE2EDuration="38.776233407s" totalImagesPullingTime="30.162343786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:26.774154302 +0000 UTC m=+39.715458149" watchObservedRunningTime="2026-07-06 20:35:26.776233407 +0000 UTC m=+39.717537254"
Jul 06 20:35:27.742637 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:27.742464    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-57cbl" event={"ID":"ec945c9f-5716-49f2-8dd7-0347fe6275dd","Type":"ContainerStarted","Data":"9a43ed32e4a96889ac97791cf662a5ee3acbcaf0b594fd553fe3827515f43678"}
Jul 06 20:35:27.768710 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:27.768666    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-57cbl" podStartSLOduration=2.98926398 podStartE2EDuration="5.768650593s" totalImagesPullingTime="2.779386613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:27.768466932 +0000 UTC m=+40.709770779" watchObservedRunningTime="2026-07-06 20:35:27.768650593 +0000 UTC m=+40.709954673"
Jul 06 20:35:30.298918 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:30.298885    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:30.299271 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:30.299033    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:35:30.299271 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:30.299048    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7558884f8-xtl65: secret "image-registry-tls" not found
Jul 06 20:35:30.299271 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:30.299114    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls podName:354a41da-1c7d-4c43-a17c-bcea3ede2a96 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:38.2990976 +0000 UTC m=+51.240401427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls") pod "image-registry-7558884f8-xtl65" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96") : secret "image-registry-tls" not found
Jul 06 20:35:30.399522 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:30.399490    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:30.399659 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:30.399601    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:30.399659 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:30.399657    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls podName:3c6624a7-c191-44b2-806f-c1a04a48402e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:38.399643635 +0000 UTC m=+51.340947474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls") pod "dns-default-nb4m7" (UID: "3c6624a7-c191-44b2-806f-c1a04a48402e") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:38.355021 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:38.354990    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:38.355391 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:38.355128    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:35:38.355391 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:38.355146    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7558884f8-xtl65: secret "image-registry-tls" not found
Jul 06 20:35:38.355391 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:38.355199    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls podName:354a41da-1c7d-4c43-a17c-bcea3ede2a96 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:54.355184702 +0000 UTC m=+67.296488543 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls") pod "image-registry-7558884f8-xtl65" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96") : secret "image-registry-tls" not found
Jul 06 20:35:38.455483 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:38.455455    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:38.455623 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:38.455560    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:38.455623 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:38.455610    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls podName:3c6624a7-c191-44b2-806f-c1a04a48402e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:54.455593447 +0000 UTC m=+67.396897274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls") pod "dns-default-nb4m7" (UID: "3c6624a7-c191-44b2-806f-c1a04a48402e") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:48.716321 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:48.716296    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v7kmn"
Jul 06 20:35:53.158027 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.157978    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:35:53.158481 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.158043    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:53.160559 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.160538    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:35:53.160693 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.160676    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:35:53.168637 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:53.168620    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 20:35:53.168697 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:53.168672    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs podName:9866e3e7-f454-42b7-add8-3bcf2b9f797a nodeName:}" failed. No retries permitted until 2026-07-06 20:36:57.168657663 +0000 UTC m=+130.109961502 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs") pod "network-metrics-daemon-6vs4n" (UID: "9866e3e7-f454-42b7-add8-3bcf2b9f797a") : secret "metrics-daemon-secret" not found
Jul 06 20:35:53.171987 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.171964    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6de51-6683-4b89-b584-0cb22a3b8833-original-pull-secret\") pod \"global-pull-secret-syncer-dnd82\" (UID: \"1cd6de51-6683-4b89-b584-0cb22a3b8833\") " pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:53.188118 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.188098    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnd82"
Jul 06 20:35:53.309837 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.309792    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dnd82"]
Jul 06 20:35:53.312967 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:35:53.312939    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd6de51_6683_4b89_b584_0cb22a3b8833.slice/crio-4f76ae0e77d1e8838a151c67bf9ba49289276471c4a2d6c3fb63615212930406 WatchSource:0}: Error finding container 4f76ae0e77d1e8838a151c67bf9ba49289276471c4a2d6c3fb63615212930406: Status 404 returned error can't find the container with id 4f76ae0e77d1e8838a151c67bf9ba49289276471c4a2d6c3fb63615212930406
Jul 06 20:35:53.360079 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.360052    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248l8\" (UniqueName: \"kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8\") pod \"network-check-target-kkslg\" (UID: \"7510d9cd-07f1-4c52-ad19-3a31394c9300\") " pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:53.362859 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.362841    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:35:53.373820 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.373794    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:35:53.384314 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.384292    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248l8\" (UniqueName: \"kubernetes.io/projected/7510d9cd-07f1-4c52-ad19-3a31394c9300-kube-api-access-248l8\") pod \"network-check-target-kkslg\" (UID: \"7510d9cd-07f1-4c52-ad19-3a31394c9300\") " pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:53.497242 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.497193    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6bf8b\""
Jul 06 20:35:53.505265 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.505253    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:53.622569 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.622490    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kkslg"]
Jul 06 20:35:53.624741 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:35:53.624716    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7510d9cd_07f1_4c52_ad19_3a31394c9300.slice/crio-6f0edade64f392f0153131f8134410914ceef61390feebeb09c292d54cd76872 WatchSource:0}: Error finding container 6f0edade64f392f0153131f8134410914ceef61390feebeb09c292d54cd76872: Status 404 returned error can't find the container with id 6f0edade64f392f0153131f8134410914ceef61390feebeb09c292d54cd76872
Jul 06 20:35:53.803521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.803439    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kkslg" event={"ID":"7510d9cd-07f1-4c52-ad19-3a31394c9300","Type":"ContainerStarted","Data":"6f0edade64f392f0153131f8134410914ceef61390feebeb09c292d54cd76872"}
Jul 06 20:35:53.804653 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:53.804620    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dnd82" event={"ID":"1cd6de51-6683-4b89-b584-0cb22a3b8833","Type":"ContainerStarted","Data":"4f76ae0e77d1e8838a151c67bf9ba49289276471c4a2d6c3fb63615212930406"}
Jul 06 20:35:54.366637 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:54.366592    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:35:54.367104 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:54.366742    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:35:54.367104 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:54.366757    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7558884f8-xtl65: secret "image-registry-tls" not found
Jul 06 20:35:54.367104 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:54.366846    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls podName:354a41da-1c7d-4c43-a17c-bcea3ede2a96 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:26.366827039 +0000 UTC m=+99.308130878 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls") pod "image-registry-7558884f8-xtl65" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96") : secret "image-registry-tls" not found
Jul 06 20:35:54.467836 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:54.467765    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:35:54.468005 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:54.467911    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:54.468005 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:54.467978    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls podName:3c6624a7-c191-44b2-806f-c1a04a48402e nodeName:}" failed. No retries permitted until 2026-07-06 20:36:26.467957322 +0000 UTC m=+99.409261161 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls") pod "dns-default-nb4m7" (UID: "3c6624a7-c191-44b2-806f-c1a04a48402e") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:54.936443 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:54.936407    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"]
Jul 06 20:35:54.957046 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:54.956357    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:54.957972 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:54.957947    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"]
Jul 06 20:35:54.959997 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:54.959972    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mgg7v\""
Jul 06 20:35:54.960257 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:54.960187    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 06 20:35:54.960543 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:54.960522    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 06 20:35:54.960623 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:54.960562    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:54.960828 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:54.960797    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 06 20:35:55.032464 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.032430    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"]
Jul 06 20:35:55.056225 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.056201    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"]
Jul 06 20:35:55.056383 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.056317    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:35:55.059949 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.059924    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p7z6j\""
Jul 06 20:35:55.060223 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.060176    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 06 20:35:55.060349 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.060187    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:55.060749 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.060573    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 06 20:35:55.072149 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.072113    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eaf12a17-0dd7-4e9d-939f-33442530321f-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8f2m8\" (UID: \"eaf12a17-0dd7-4e9d-939f-33442530321f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:55.072249 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.072231    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eaf12a17-0dd7-4e9d-939f-33442530321f-config\") pod \"service-ca-operator-5d46d9ffd4-8f2m8\" (UID: \"eaf12a17-0dd7-4e9d-939f-33442530321f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:55.072311 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.072291    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqgxm\" (UniqueName: \"kubernetes.io/projected/eaf12a17-0dd7-4e9d-939f-33442530321f-kube-api-access-hqgxm\") pod \"service-ca-operator-5d46d9ffd4-8f2m8\" (UID: \"eaf12a17-0dd7-4e9d-939f-33442530321f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:55.172890 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.172854    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqgxm\" (UniqueName: \"kubernetes.io/projected/eaf12a17-0dd7-4e9d-939f-33442530321f-kube-api-access-hqgxm\") pod \"service-ca-operator-5d46d9ffd4-8f2m8\" (UID: \"eaf12a17-0dd7-4e9d-939f-33442530321f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:55.173067 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.172909    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eaf12a17-0dd7-4e9d-939f-33442530321f-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8f2m8\" (UID: \"eaf12a17-0dd7-4e9d-939f-33442530321f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:55.173067 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.173009    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eaf12a17-0dd7-4e9d-939f-33442530321f-config\") pod \"service-ca-operator-5d46d9ffd4-8f2m8\" (UID: \"eaf12a17-0dd7-4e9d-939f-33442530321f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:55.173067 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.173043    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:35:55.173226 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.173080    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8wq\" (UniqueName: \"kubernetes.io/projected/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-kube-api-access-2b8wq\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:35:55.173797 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.173749    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eaf12a17-0dd7-4e9d-939f-33442530321f-config\") pod \"service-ca-operator-5d46d9ffd4-8f2m8\" (UID: \"eaf12a17-0dd7-4e9d-939f-33442530321f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:55.176180 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.176153    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eaf12a17-0dd7-4e9d-939f-33442530321f-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8f2m8\" (UID: \"eaf12a17-0dd7-4e9d-939f-33442530321f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:55.184342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.184320    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqgxm\" (UniqueName: \"kubernetes.io/projected/eaf12a17-0dd7-4e9d-939f-33442530321f-kube-api-access-hqgxm\") pod \"service-ca-operator-5d46d9ffd4-8f2m8\" (UID: \"eaf12a17-0dd7-4e9d-939f-33442530321f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:55.268772 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.268703    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"
Jul 06 20:35:55.274044 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.274021    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:35:55.274155 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.274061    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8wq\" (UniqueName: \"kubernetes.io/projected/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-kube-api-access-2b8wq\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:35:55.274210 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:55.274175    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:35:55.274254 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:55.274241    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls podName:2bc4f28b-6850-43a0-b4e1-b6fb9c76f605 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:55.774222453 +0000 UTC m=+68.715526283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7vfgj" (UID: "2bc4f28b-6850-43a0-b4e1-b6fb9c76f605") : secret "samples-operator-tls" not found
Jul 06 20:35:55.282891 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.282866    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8wq\" (UniqueName: \"kubernetes.io/projected/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-kube-api-access-2b8wq\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:35:55.436892 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.436825    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8"]
Jul 06 20:35:55.440537 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:35:55.440448    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf12a17_0dd7_4e9d_939f_33442530321f.slice/crio-882a2365feb21aff2858b8769068980a0c1f380cc245714bcc67278601291582 WatchSource:0}: Error finding container 882a2365feb21aff2858b8769068980a0c1f380cc245714bcc67278601291582: Status 404 returned error can't find the container with id 882a2365feb21aff2858b8769068980a0c1f380cc245714bcc67278601291582
Jul 06 20:35:55.777781 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.777730    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:35:55.777958 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:55.777900    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:35:55.778013 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:55.777972    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls podName:2bc4f28b-6850-43a0-b4e1-b6fb9c76f605 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:56.777955602 +0000 UTC m=+69.719259432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7vfgj" (UID: "2bc4f28b-6850-43a0-b4e1-b6fb9c76f605") : secret "samples-operator-tls" not found
Jul 06 20:35:55.811168 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:55.811126    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8" event={"ID":"eaf12a17-0dd7-4e9d-939f-33442530321f","Type":"ContainerStarted","Data":"882a2365feb21aff2858b8769068980a0c1f380cc245714bcc67278601291582"}
Jul 06 20:35:56.784822 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:56.784766    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:35:56.785203 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:56.784931    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:35:56.785203 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:56.785006    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls podName:2bc4f28b-6850-43a0-b4e1-b6fb9c76f605 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:58.78498422 +0000 UTC m=+71.726288060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7vfgj" (UID: "2bc4f28b-6850-43a0-b4e1-b6fb9c76f605") : secret "samples-operator-tls" not found
Jul 06 20:35:58.800344 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:58.800310    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:35:58.801050 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:58.800441    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:35:58.801050 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:58.800511    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls podName:2bc4f28b-6850-43a0-b4e1-b6fb9c76f605 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:02.800492604 +0000 UTC m=+75.741796434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7vfgj" (UID: "2bc4f28b-6850-43a0-b4e1-b6fb9c76f605") : secret "samples-operator-tls" not found
Jul 06 20:35:58.820086 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:58.820051    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dnd82" event={"ID":"1cd6de51-6683-4b89-b584-0cb22a3b8833","Type":"ContainerStarted","Data":"09231ffd08dccddd6c16ac4984b2a094d6168bef1f86b7794f3af3165ff71d3c"}
Jul 06 20:35:58.821672 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:58.821640    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kkslg" event={"ID":"7510d9cd-07f1-4c52-ad19-3a31394c9300","Type":"ContainerStarted","Data":"ef591ed513720af34edbdfb5b0d9f9fede5ea89fcdf8c104f89a9b5b2dfe5946"}
Jul 06 20:35:58.821783 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:58.821691    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:35:58.856385 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:58.856330    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dnd82" podStartSLOduration=66.151869169 podStartE2EDuration="1m10.856310106s" totalImagesPullingTime="4.704440937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:58.855024918 +0000 UTC m=+71.796328767" watchObservedRunningTime="2026-07-06 20:35:58.856310106 +0000 UTC m=+71.797613955"
Jul 06 20:35:58.881211 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:58.881173    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kkslg" podStartSLOduration=66.494114866 podStartE2EDuration="1m10.88115967s" totalImagesPullingTime="4.387044804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:58.880438887 +0000 UTC m=+71.821742734" watchObservedRunningTime="2026-07-06 20:35:58.88115967 +0000 UTC m=+71.822463527"
Jul 06 20:35:59.274037 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.274000    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4ct7m"]
Jul 06 20:35:59.277024 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.277001    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.280644 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.280622    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 20:35:59.280907 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.280886    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 20:35:59.281159 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.281139    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 20:35:59.281999 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.281978    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 20:35:59.282079 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.281980    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-42v4d\""
Jul 06 20:35:59.293662 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.293639    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4ct7m"]
Jul 06 20:35:59.405782 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.405746    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.405957 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.405788    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.405957 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.405882    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rv9b\" (UniqueName: \"kubernetes.io/projected/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-api-access-2rv9b\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.405957 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.405906    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-crio-socket\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.406102 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.406080    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-data-volume\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.506957 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.506925    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-data-volume\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.507132 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.506966    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.507132 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.506990    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.507132 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.507036    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rv9b\" (UniqueName: \"kubernetes.io/projected/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-api-access-2rv9b\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.507132 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.507062    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-crio-socket\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.507132 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:59.507061    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:35:59.507132 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:59.507112    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:35:59.507132 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:59.507137    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:00.007117338 +0000 UTC m=+72.948421180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : configmap "kube-rbac-proxy" not found
Jul 06 20:35:59.507474 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:35:59.507155    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:00.007143511 +0000 UTC m=+72.948447347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:35:59.507474 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.507171    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-crio-socket\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.507474 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.507310    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-data-volume\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:35:59.519440 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:35:59.519419    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rv9b\" (UniqueName: \"kubernetes.io/projected/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-api-access-2rv9b\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:00.011119 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:00.011076    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:00.011486 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:00.011131    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:00.011486 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:00.011209    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:36:00.011486 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:00.011283    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:01.01126503 +0000 UTC m=+73.952568856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : configmap "kube-rbac-proxy" not found
Jul 06 20:36:00.011486 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:00.011294    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:00.011486 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:00.011346    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:01.011333121 +0000 UTC m=+73.952636945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:00.827572 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:00.827533    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8" event={"ID":"eaf12a17-0dd7-4e9d-939f-33442530321f","Type":"ContainerStarted","Data":"954a79d2b2ab7f0267b21c82ee17e743eeb8b6b9adb3bfead269948ab741fb90"}
Jul 06 20:36:00.847086 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:00.847044    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8" podStartSLOduration=2.436779317 podStartE2EDuration="6.847029631s" totalImagesPullingTime="4.410250314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:00.846068289 +0000 UTC m=+73.787372135" watchObservedRunningTime="2026-07-06 20:36:00.847029631 +0000 UTC m=+73.788333477"
Jul 06 20:36:01.018708 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:01.018679    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:01.018708 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:01.018711    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:01.019098 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:01.018789    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:36:01.019098 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:01.018823    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:01.019098 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:01.018859    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:03.018846589 +0000 UTC m=+75.960150419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : configmap "kube-rbac-proxy" not found
Jul 06 20:36:01.019098 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:01.018874    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:03.018867815 +0000 UTC m=+75.960171639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:01.996744 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:01.996719    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fp74t_5eef1e47-1131-4820-b054-5ad627772e0f/dns-node-resolver/0.log"
Jul 06 20:36:02.834177 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:02.834150    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:36:02.834562 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:02.834254    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:36:02.834562 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:02.834303    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls podName:2bc4f28b-6850-43a0-b4e1-b6fb9c76f605 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:10.834289981 +0000 UTC m=+83.775593806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7vfgj" (UID: "2bc4f28b-6850-43a0-b4e1-b6fb9c76f605") : secret "samples-operator-tls" not found
Jul 06 20:36:03.036242 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:03.036209    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:03.036242 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:03.036241    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:03.036521 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:03.036353    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:36:03.036521 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:03.036374    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:03.036521 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:03.036421    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:07.036406554 +0000 UTC m=+79.977710380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:03.036521 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:03.036434    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:07.036427583 +0000 UTC m=+79.977731407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : configmap "kube-rbac-proxy" not found
Jul 06 20:36:03.197798 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:03.197729    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mqszb_6fe996a8-9939-4be3-a9d2-a574dbc4ce24/node-ca/0.log"
Jul 06 20:36:03.592492 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:03.592470    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-57cbl_ec945c9f-5716-49f2-8dd7-0347fe6275dd/serve-healthcheck-canary/0.log"
Jul 06 20:36:07.065402 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:07.065363    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:07.065402 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:07.065404    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:07.065846 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:07.065507    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:36:07.065846 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:07.065574    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:15.065558241 +0000 UTC m=+88.006862069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : configmap "kube-rbac-proxy" not found
Jul 06 20:36:07.065846 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:07.065578    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:07.065846 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:07.065635    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:15.065618535 +0000 UTC m=+88.006922359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:10.890322 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:10.890202    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:36:10.890698 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:10.890351    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:36:10.890698 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:10.890409    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls podName:2bc4f28b-6850-43a0-b4e1-b6fb9c76f605 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:26.890393943 +0000 UTC m=+99.831697767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7vfgj" (UID: "2bc4f28b-6850-43a0-b4e1-b6fb9c76f605") : secret "samples-operator-tls" not found
Jul 06 20:36:15.123904 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:15.123869    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:15.123904 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:15.123904    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:15.124303 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:15.124017    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:15.124303 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:15.124099    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls podName:1f3b2b13-dcaa-4ce6-8e00-201340f91ed0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:31.124080091 +0000 UTC m=+104.065383936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ct7m" (UID: "1f3b2b13-dcaa-4ce6-8e00-201340f91ed0") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:15.124482 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:15.124457    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:26.409124 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.409083    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:36:26.411388 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.411356    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"image-registry-7558884f8-xtl65\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") " pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:36:26.438382 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.438359    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nc576\""
Jul 06 20:36:26.446387 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.446371    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:36:26.510404 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.510370    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:36:26.512975 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.512947    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c6624a7-c191-44b2-806f-c1a04a48402e-metrics-tls\") pod \"dns-default-nb4m7\" (UID: \"3c6624a7-c191-44b2-806f-c1a04a48402e\") " pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:36:26.572165 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.572074    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7558884f8-xtl65"]
Jul 06 20:36:26.574613 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:26.574583    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354a41da_1c7d_4c43_a17c_bcea3ede2a96.slice/crio-ac3079de51e8c1f41797d68f2dc570eb684d4f089e61f656728427c7794a05d7 WatchSource:0}: Error finding container ac3079de51e8c1f41797d68f2dc570eb684d4f089e61f656728427c7794a05d7: Status 404 returned error can't find the container with id ac3079de51e8c1f41797d68f2dc570eb684d4f089e61f656728427c7794a05d7
Jul 06 20:36:26.770123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.770054    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dqcct\""
Jul 06 20:36:26.778345 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.778324    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:36:26.896088 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.896056    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7558884f8-xtl65" event={"ID":"354a41da-1c7d-4c43-a17c-bcea3ede2a96","Type":"ContainerStarted","Data":"00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26"}
Jul 06 20:36:26.896249 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.896108    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7558884f8-xtl65" event={"ID":"354a41da-1c7d-4c43-a17c-bcea3ede2a96","Type":"ContainerStarted","Data":"ac3079de51e8c1f41797d68f2dc570eb684d4f089e61f656728427c7794a05d7"}
Jul 06 20:36:26.896249 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.896194    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:36:26.902728 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.902706    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nb4m7"]
Jul 06 20:36:26.905929 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:26.905907    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6624a7_c191_44b2_806f_c1a04a48402e.slice/crio-5b0464fb408322e3034c146609cfa96623b6a464282acc639c378e9937c4a251 WatchSource:0}: Error finding container 5b0464fb408322e3034c146609cfa96623b6a464282acc639c378e9937c4a251: Status 404 returned error can't find the container with id 5b0464fb408322e3034c146609cfa96623b6a464282acc639c378e9937c4a251
Jul 06 20:36:26.913503 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.913484    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:36:26.915715 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.915695    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f28b-6850-43a0-b4e1-b6fb9c76f605-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7vfgj\" (UID: \"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:36:26.921871 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:26.921832    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7558884f8-xtl65" podStartSLOduration=98.921803477 podStartE2EDuration="1m38.921803477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:26.92089685 +0000 UTC m=+99.862200699" watchObservedRunningTime="2026-07-06 20:36:26.921803477 +0000 UTC m=+99.863107324"
Jul 06 20:36:27.168554 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.168523    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"
Jul 06 20:36:27.290719 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.290696    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj"]
Jul 06 20:36:27.900594 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.900543    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj" event={"ID":"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605","Type":"ContainerStarted","Data":"99c1d84f9d6756154d80b2d4dff2035351ecb98d8f6c162913ede1d05cb875a8"}
Jul 06 20:36:27.902471 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.902440    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nb4m7" event={"ID":"3c6624a7-c191-44b2-806f-c1a04a48402e","Type":"ContainerStarted","Data":"5b0464fb408322e3034c146609cfa96623b6a464282acc639c378e9937c4a251"}
Jul 06 20:36:27.956375 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.956340    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"]
Jul 06 20:36:27.978295 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.978259    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"]
Jul 06 20:36:27.978469 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.978379    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:27.983473 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.983448    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 20:36:27.983616 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.983574    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 20:36:27.983690 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.983623    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 20:36:27.983690 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:27.983685    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 06 20:36:28.091512 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.091486    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-qkh89"]
Jul 06 20:36:28.114215 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.114174    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-qkh89"]
Jul 06 20:36:28.114368 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.114264    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-qkh89"
Jul 06 20:36:28.118448 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.118330    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 06 20:36:28.118564 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.118538    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 06 20:36:28.118626 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.118610    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ftjvm\""
Jul 06 20:36:28.123160 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.123113    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zpb\" (UniqueName: \"kubernetes.io/projected/27c84ec1-53f3-431e-89ff-aa0e9e1e8d41-kube-api-access-54zpb\") pod \"klusterlet-addon-workmgr-7b4d685f66-hqrgp\" (UID: \"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:28.123275 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.123231    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27c84ec1-53f3-431e-89ff-aa0e9e1e8d41-tmp\") pod \"klusterlet-addon-workmgr-7b4d685f66-hqrgp\" (UID: \"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:28.123349 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.123274    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/27c84ec1-53f3-431e-89ff-aa0e9e1e8d41-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b4d685f66-hqrgp\" (UID: \"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:28.224100 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.224015    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/27c84ec1-53f3-431e-89ff-aa0e9e1e8d41-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b4d685f66-hqrgp\" (UID: \"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:28.224100 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.224095    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56r2t\" (UniqueName: \"kubernetes.io/projected/6c4dcd10-1f2f-4405-805a-9694a2122c6f-kube-api-access-56r2t\") pod \"downloads-685bfb5f64-qkh89\" (UID: \"6c4dcd10-1f2f-4405-805a-9694a2122c6f\") " pod="openshift-console/downloads-685bfb5f64-qkh89"
Jul 06 20:36:28.224290 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.224227    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zpb\" (UniqueName: \"kubernetes.io/projected/27c84ec1-53f3-431e-89ff-aa0e9e1e8d41-kube-api-access-54zpb\") pod \"klusterlet-addon-workmgr-7b4d685f66-hqrgp\" (UID: \"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:28.224328 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.224293    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27c84ec1-53f3-431e-89ff-aa0e9e1e8d41-tmp\") pod \"klusterlet-addon-workmgr-7b4d685f66-hqrgp\" (UID: \"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:28.224693 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.224648    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27c84ec1-53f3-431e-89ff-aa0e9e1e8d41-tmp\") pod \"klusterlet-addon-workmgr-7b4d685f66-hqrgp\" (UID: \"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:28.226973 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.226949    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/27c84ec1-53f3-431e-89ff-aa0e9e1e8d41-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b4d685f66-hqrgp\" (UID: \"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:28.248284 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.248251    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zpb\" (UniqueName: \"kubernetes.io/projected/27c84ec1-53f3-431e-89ff-aa0e9e1e8d41-kube-api-access-54zpb\") pod \"klusterlet-addon-workmgr-7b4d685f66-hqrgp\" (UID: \"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:28.291082 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.291051    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:28.325519 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.325491    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56r2t\" (UniqueName: \"kubernetes.io/projected/6c4dcd10-1f2f-4405-805a-9694a2122c6f-kube-api-access-56r2t\") pod \"downloads-685bfb5f64-qkh89\" (UID: \"6c4dcd10-1f2f-4405-805a-9694a2122c6f\") " pod="openshift-console/downloads-685bfb5f64-qkh89"
Jul 06 20:36:28.345211 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.345187    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56r2t\" (UniqueName: \"kubernetes.io/projected/6c4dcd10-1f2f-4405-805a-9694a2122c6f-kube-api-access-56r2t\") pod \"downloads-685bfb5f64-qkh89\" (UID: \"6c4dcd10-1f2f-4405-805a-9694a2122c6f\") " pod="openshift-console/downloads-685bfb5f64-qkh89"
Jul 06 20:36:28.426226 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:28.426187    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-qkh89"
Jul 06 20:36:29.356083 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.356053    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4"]
Jul 06 20:36:29.377769 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.377746    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4"
Jul 06 20:36:29.377904 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.377748    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4"]
Jul 06 20:36:29.380696 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.380678    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 06 20:36:29.380913 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.380898    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-689mp\""
Jul 06 20:36:29.534234 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.534213    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/370f67b9-b322-455b-b7df-099c1b5750f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-djgk4\" (UID: \"370f67b9-b322-455b-b7df-099c1b5750f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4"
Jul 06 20:36:29.634667 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.634605    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/370f67b9-b322-455b-b7df-099c1b5750f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-djgk4\" (UID: \"370f67b9-b322-455b-b7df-099c1b5750f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4"
Jul 06 20:36:29.636931 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.636910    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/370f67b9-b322-455b-b7df-099c1b5750f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-djgk4\" (UID: \"370f67b9-b322-455b-b7df-099c1b5750f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4"
Jul 06 20:36:29.686885 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.686865    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4"
Jul 06 20:36:29.833425 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.833120    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kkslg"
Jul 06 20:36:29.945591 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.945488    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"]
Jul 06 20:36:29.961328 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.961310    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-qkh89"]
Jul 06 20:36:29.963947 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:29.963892    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4dcd10_1f2f_4405_805a_9694a2122c6f.slice/crio-40baa6bff0b59980dd04067f65a1a2328124f7a0279236f002a6f353e577e793 WatchSource:0}: Error finding container 40baa6bff0b59980dd04067f65a1a2328124f7a0279236f002a6f353e577e793: Status 404 returned error can't find the container with id 40baa6bff0b59980dd04067f65a1a2328124f7a0279236f002a6f353e577e793
Jul 06 20:36:29.977099 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:29.977049    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4"]
Jul 06 20:36:29.982430 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:29.982097    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370f67b9_b322_455b_b7df_099c1b5750f5.slice/crio-90596191390e3e2657f7dad3efaadc9e28d85c9fa83ce43f0f51cf91d67dbe84 WatchSource:0}: Error finding container 90596191390e3e2657f7dad3efaadc9e28d85c9fa83ce43f0f51cf91d67dbe84: Status 404 returned error can't find the container with id 90596191390e3e2657f7dad3efaadc9e28d85c9fa83ce43f0f51cf91d67dbe84
Jul 06 20:36:30.917736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:30.917694    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj" event={"ID":"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605","Type":"ContainerStarted","Data":"e74fc11eae2cbbc819299593bf0a852c876d45a5e23ad849f5cb693afc4fba08"}
Jul 06 20:36:30.918318 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:30.917764    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj" event={"ID":"2bc4f28b-6850-43a0-b4e1-b6fb9c76f605","Type":"ContainerStarted","Data":"b29d322979d5ffa65a59e22d945021fdc9ca736a8e04f4c49282a888d2b74969"}
Jul 06 20:36:30.920629 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:30.920600    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nb4m7" event={"ID":"3c6624a7-c191-44b2-806f-c1a04a48402e","Type":"ContainerStarted","Data":"30a95d8bfbbb737c3b08cbedb72f1f5f83b1ec645c7453db4a9af7d825f81ee6"}
Jul 06 20:36:30.920751 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:30.920654    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nb4m7" event={"ID":"3c6624a7-c191-44b2-806f-c1a04a48402e","Type":"ContainerStarted","Data":"8c8ed22104ef51b9b83efccd19bb8d9373cd4745bd6426deeb1b836fbddc52ec"}
Jul 06 20:36:30.921192 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:30.921171    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:36:30.922684 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:30.922656    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-qkh89" event={"ID":"6c4dcd10-1f2f-4405-805a-9694a2122c6f","Type":"ContainerStarted","Data":"40baa6bff0b59980dd04067f65a1a2328124f7a0279236f002a6f353e577e793"}
Jul 06 20:36:30.925867 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:30.925777    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4" event={"ID":"370f67b9-b322-455b-b7df-099c1b5750f5","Type":"ContainerStarted","Data":"90596191390e3e2657f7dad3efaadc9e28d85c9fa83ce43f0f51cf91d67dbe84"}
Jul 06 20:36:30.927380 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:30.927321    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp" event={"ID":"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41","Type":"ContainerStarted","Data":"d42326fd3a1b3a4f8077f95d9c8db7b46a83841b1ce36baedc3c48cc09febe4b"}
Jul 06 20:36:30.966747 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:30.966694    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7vfgj" podStartSLOduration=33.540451934000004 podStartE2EDuration="35.966676193s" totalImagesPullingTime="2.426224259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:30.940383538 +0000 UTC m=+103.881687385" watchObservedRunningTime="2026-07-06 20:36:30.966676193 +0000 UTC m=+103.907980050"
Jul 06 20:36:30.967723 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:30.967674    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nb4m7" podStartSLOduration=66.115383237 podStartE2EDuration="1m8.967663413s" totalImagesPullingTime="2.852280176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:30.966077785 +0000 UTC m=+103.907381632" watchObservedRunningTime="2026-07-06 20:36:30.967663413 +0000 UTC m=+103.908967260"
Jul 06 20:36:31.153882 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.153838    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:31.165250 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.165190    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f3b2b13-dcaa-4ce6-8e00-201340f91ed0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ct7m\" (UID: \"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0\") " pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:31.387976 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.387734    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4ct7m"
Jul 06 20:36:31.499447 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.499412    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9cb44d9bd-8hwkc"]
Jul 06 20:36:31.516516 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.515839    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.521019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.520174    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 06 20:36:31.521019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.520405    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 06 20:36:31.521019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.520565    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 06 20:36:31.521019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.520733    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n2dlm\""
Jul 06 20:36:31.521019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.520919    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 06 20:36:31.521019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.520928    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 06 20:36:31.525777 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.525752    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9cb44d9bd-8hwkc"]
Jul 06 20:36:31.576359 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.576331    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4ct7m"]
Jul 06 20:36:31.614304 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:31.614275    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3b2b13_dcaa_4ce6_8e00_201340f91ed0.slice/crio-a56d078484dfbff6b683eb10fb6f25ffaa8bbba10a32a821d17453153856f13e WatchSource:0}: Error finding container a56d078484dfbff6b683eb10fb6f25ffaa8bbba10a32a821d17453153856f13e: Status 404 returned error can't find the container with id a56d078484dfbff6b683eb10fb6f25ffaa8bbba10a32a821d17453153856f13e
Jul 06 20:36:31.658864 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.658835    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-oauth-serving-cert\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.658980 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.658918    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-config\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.659042 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.659013    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-serving-cert\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.659101 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.659041    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-oauth-config\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.659101 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.659072    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4tz\" (UniqueName: \"kubernetes.io/projected/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-kube-api-access-nd4tz\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.659203 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.659136    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-service-ca\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.760438 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.759923    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-serving-cert\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.760438 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.759974    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-oauth-config\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.760438 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.760033    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4tz\" (UniqueName: \"kubernetes.io/projected/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-kube-api-access-nd4tz\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.760438 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.760097    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-service-ca\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.760438 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.760145    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-oauth-serving-cert\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.760438 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.760163    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-config\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.760882 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.760855    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-config\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.761305 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.761277    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-oauth-serving-cert\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.761943 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.761792    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-service-ca\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.763171 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.763148    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-serving-cert\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.763268 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.763192    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-oauth-config\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.774200 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.774177    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4tz\" (UniqueName: \"kubernetes.io/projected/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-kube-api-access-nd4tz\") pod \"console-9cb44d9bd-8hwkc\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") " pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.830868 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.830828    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:31.935274 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.935195    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ct7m" event={"ID":"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0","Type":"ContainerStarted","Data":"a56d078484dfbff6b683eb10fb6f25ffaa8bbba10a32a821d17453153856f13e"}
Jul 06 20:36:31.937915 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.937779    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4" event={"ID":"370f67b9-b322-455b-b7df-099c1b5750f5","Type":"ContainerStarted","Data":"27de13b4b8324be9e51b3e5e959817825aedb102cc29b66e031ab64986f99dec"}
Jul 06 20:36:31.958547 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:31.958285    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4" podStartSLOduration=1.319660095 podStartE2EDuration="2.958264521s" totalImagesPullingTime="1.638604426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:31.956996928 +0000 UTC m=+104.898300820" watchObservedRunningTime="2026-07-06 20:36:31.958264521 +0000 UTC m=+104.899568369"
Jul 06 20:36:32.821843 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:32.821745    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9cb44d9bd-8hwkc"]
Jul 06 20:36:32.827601 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:32.827569    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4af1ffc_9cdb_46cc_a339_531fe2c544c5.slice/crio-2176923295de78b92d63d31de676bef975b08fc82a698cfb169e9b4176d250a5 WatchSource:0}: Error finding container 2176923295de78b92d63d31de676bef975b08fc82a698cfb169e9b4176d250a5: Status 404 returned error can't find the container with id 2176923295de78b92d63d31de676bef975b08fc82a698cfb169e9b4176d250a5
Jul 06 20:36:32.948045 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:32.947905    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cb44d9bd-8hwkc" event={"ID":"a4af1ffc-9cdb-46cc-a339-531fe2c544c5","Type":"ContainerStarted","Data":"2176923295de78b92d63d31de676bef975b08fc82a698cfb169e9b4176d250a5"}
Jul 06 20:36:32.948477 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:32.948417    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4"
Jul 06 20:36:32.954887 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:32.954689    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-djgk4"
Jul 06 20:36:33.436079 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.434818    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"]
Jul 06 20:36:33.438466 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.438439    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.445389 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.445369    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 20:36:33.445625 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.445610    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 20:36:33.445852 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.445825    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 06 20:36:33.446201 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.446184    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 20:36:33.446396 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.446381    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h66m5\""
Jul 06 20:36:33.447178 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.447121    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 06 20:36:33.451305 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.451076    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"]
Jul 06 20:36:33.578096 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.577928    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acb32073-6101-4721-a335-582734e6e8bc-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.578096 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.577979    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acb32073-6101-4721-a335-582734e6e8bc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.578328 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.578105    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acb32073-6101-4721-a335-582734e6e8bc-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.578328 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.578146    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g4xr\" (UniqueName: \"kubernetes.io/projected/acb32073-6101-4721-a335-582734e6e8bc-kube-api-access-6g4xr\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.679070 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.679025    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acb32073-6101-4721-a335-582734e6e8bc-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.679229 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.679075    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acb32073-6101-4721-a335-582734e6e8bc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.679229 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.679131    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acb32073-6101-4721-a335-582734e6e8bc-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.679229 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.679152    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g4xr\" (UniqueName: \"kubernetes.io/projected/acb32073-6101-4721-a335-582734e6e8bc-kube-api-access-6g4xr\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.680128 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.680083    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acb32073-6101-4721-a335-582734e6e8bc-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.683304 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.683278    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acb32073-6101-4721-a335-582734e6e8bc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.686568 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.686494    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acb32073-6101-4721-a335-582734e6e8bc-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.692957 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.690146    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g4xr\" (UniqueName: \"kubernetes.io/projected/acb32073-6101-4721-a335-582734e6e8bc-kube-api-access-6g4xr\") pod \"prometheus-operator-567f7b8b7c-mnpck\" (UID: \"acb32073-6101-4721-a335-582734e6e8bc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:33.755282 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:33.755222    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"
Jul 06 20:36:37.588232 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:37.587957    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck"]
Jul 06 20:36:38.706068 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:38.706035    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb32073_6101_4721_a335_582734e6e8bc.slice/crio-6429b4ac3e2757bfba32ee779e1d40ca7a5169a3c852c0e101a787a0b0b6e782 WatchSource:0}: Error finding container 6429b4ac3e2757bfba32ee779e1d40ca7a5169a3c852c0e101a787a0b0b6e782: Status 404 returned error can't find the container with id 6429b4ac3e2757bfba32ee779e1d40ca7a5169a3c852c0e101a787a0b0b6e782
Jul 06 20:36:38.973930 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:38.973826    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ct7m" event={"ID":"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0","Type":"ContainerStarted","Data":"365f8b5745542669f3351be1cb9f3afaa0807bf8f39b7f46bab62c4860f4e843"}
Jul 06 20:36:38.975573 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:38.975516    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cb44d9bd-8hwkc" event={"ID":"a4af1ffc-9cdb-46cc-a339-531fe2c544c5","Type":"ContainerStarted","Data":"08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233"}
Jul 06 20:36:38.976866 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:38.976839    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck" event={"ID":"acb32073-6101-4721-a335-582734e6e8bc","Type":"ContainerStarted","Data":"6429b4ac3e2757bfba32ee779e1d40ca7a5169a3c852c0e101a787a0b0b6e782"}
Jul 06 20:36:38.979154 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:38.979110    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp" event={"ID":"27c84ec1-53f3-431e-89ff-aa0e9e1e8d41","Type":"ContainerStarted","Data":"2f5f518a8578afadc61a26f690e92332e7fd198735c2792f1cadf86d46420a32"}
Jul 06 20:36:38.979529 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:38.979509    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:38.981030 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:38.980979    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp"
Jul 06 20:36:39.001429 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:39.001388    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9cb44d9bd-8hwkc" podStartSLOduration=2.085442199 podStartE2EDuration="8.001370165s" totalImagesPullingTime="5.915927966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:38.999681748 +0000 UTC m=+111.940985596" watchObservedRunningTime="2026-07-06 20:36:39.001370165 +0000 UTC m=+111.942674016"
Jul 06 20:36:39.019971 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:39.019936    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b4d685f66-hqrgp" podStartSLOduration=4.523404555 podStartE2EDuration="12.019923636s" totalImagesPullingTime="7.496519081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:39.019143013 +0000 UTC m=+111.960446871" watchObservedRunningTime="2026-07-06 20:36:39.019923636 +0000 UTC m=+111.961227483"
Jul 06 20:36:39.985413 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:39.985324    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ct7m" event={"ID":"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0","Type":"ContainerStarted","Data":"394355572b6bed8d74abf3757cfc5edd99eebe81e85a34bca12c2bead85cb030"}
Jul 06 20:36:40.991666 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:40.991630    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck" event={"ID":"acb32073-6101-4721-a335-582734e6e8bc","Type":"ContainerStarted","Data":"40bc7c726680eb2c3833a2906c34237e767f4bb699dc3940b8737dd659f527db"}
Jul 06 20:36:40.992145 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:40.991693    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck" event={"ID":"acb32073-6101-4721-a335-582734e6e8bc","Type":"ContainerStarted","Data":"ba6ff343ad9199b112bc084ce252e76187181036fb516f91e46b1275ce30318a"}
Jul 06 20:36:41.015843 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:41.015770    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mnpck" podStartSLOduration=6.268530918 podStartE2EDuration="8.015749358s" totalImagesPullingTime="1.74721844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:41.013126421 +0000 UTC m=+113.954430268" watchObservedRunningTime="2026-07-06 20:36:41.015749358 +0000 UTC m=+113.957053208"
Jul 06 20:36:41.831206 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:41.831161    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:41.831402 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:41.831235    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:36:41.832863 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:41.832830    2550 patch_prober.go:28] interesting pod/console-9cb44d9bd-8hwkc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.15:8443/health\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body=
Jul 06 20:36:41.833016 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:41.832890    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-9cb44d9bd-8hwkc" podUID="a4af1ffc-9cdb-46cc-a339-531fe2c544c5" containerName="console" probeResult="failure" output="Get \"https://10.132.0.15:8443/health\": dial tcp 10.132.0.15:8443: connect: connection refused"
Jul 06 20:36:41.951066 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:41.951034    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nb4m7"
Jul 06 20:36:42.839347 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.839316    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"]
Jul 06 20:36:42.842920 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.842898    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.845728 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.845705    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 06 20:36:42.845897 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.845875    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-x8d84\""
Jul 06 20:36:42.846000 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.845986    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 06 20:36:42.855505 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.855485    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"]
Jul 06 20:36:42.861895 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.861861    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69c4ee10-5ebb-44d8-9552-da4996f18848-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.861996 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.861938    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69c4ee10-5ebb-44d8-9552-da4996f18848-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.861996 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.861974    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9pd\" (UniqueName: \"kubernetes.io/projected/69c4ee10-5ebb-44d8-9552-da4996f18848-kube-api-access-ql9pd\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.862109 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.862078    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69c4ee10-5ebb-44d8-9552-da4996f18848-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.899577 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.899544    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"]
Jul 06 20:36:42.902781 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.902761    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sldgn"]
Jul 06 20:36:42.902917 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.902874    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:42.906137 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.906101    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:42.906137 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.906111    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 06 20:36:42.906329 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.906247    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 06 20:36:42.906329 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.906247    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 06 20:36:42.906329 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.906117    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5mfdg\""
Jul 06 20:36:42.909352 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.909197    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 20:36:42.909749 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.909727    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 20:36:42.909886 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.909775    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gl26r\""
Jul 06 20:36:42.909954 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.909882    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 20:36:42.921496 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.921479    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"]
Jul 06 20:36:42.962547 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.962509    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:42.962547 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.962547    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:42.962761 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.962584    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69c4ee10-5ebb-44d8-9552-da4996f18848-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.962761 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.962614    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:42.962761 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.962674    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mwvz\" (UniqueName: \"kubernetes.io/projected/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-api-access-6mwvz\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:42.962761 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:42.962741    2550 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 06 20:36:42.962761 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.962741    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69c4ee10-5ebb-44d8-9552-da4996f18848-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.963025 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:42.962858    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69c4ee10-5ebb-44d8-9552-da4996f18848-openshift-state-metrics-tls podName:69c4ee10-5ebb-44d8-9552-da4996f18848 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:43.462837949 +0000 UTC m=+116.404141792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/69c4ee10-5ebb-44d8-9552-da4996f18848-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-45gjw" (UID: "69c4ee10-5ebb-44d8-9552-da4996f18848") : secret "openshift-state-metrics-tls" not found
Jul 06 20:36:42.963025 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.962911    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:42.963025 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.962955    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:42.963190 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.963031    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69c4ee10-5ebb-44d8-9552-da4996f18848-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.963190 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.963064    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9pd\" (UniqueName: \"kubernetes.io/projected/69c4ee10-5ebb-44d8-9552-da4996f18848-kube-api-access-ql9pd\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.963725 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.963708    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69c4ee10-5ebb-44d8-9552-da4996f18848-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.965584 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.965563    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69c4ee10-5ebb-44d8-9552-da4996f18848-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:42.976827 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:42.976790    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9pd\" (UniqueName: \"kubernetes.io/projected/69c4ee10-5ebb-44d8-9552-da4996f18848-kube-api-access-ql9pd\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:43.063463 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063415    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-sys\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.063463 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063458    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-textfile\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.063709 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063482    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-accelerators-collector-config\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.063709 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063527    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xrg\" (UniqueName: \"kubernetes.io/projected/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-kube-api-access-b4xrg\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.063709 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063580    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-root\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.063709 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063617    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.063709 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063682    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.063971 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063729    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.063971 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063788    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.063971 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063838    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mwvz\" (UniqueName: \"kubernetes.io/projected/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-api-access-6mwvz\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.063971 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063883    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.063971 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:43.063891    2550 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 06 20:36:43.063971 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063911    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-tls\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.063971 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:43.063956    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-tls podName:343a6f78-6839-47a4-9a6c-4f71f52aa9f0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:43.563935335 +0000 UTC m=+116.505239179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-v2h4j" (UID: "343a6f78-6839-47a4-9a6c-4f71f52aa9f0") : secret "kube-state-metrics-tls" not found
Jul 06 20:36:43.063971 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063942    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.064391 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.063987    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-metrics-client-ca\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.064391 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.064028    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-wtmp\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.064391 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.064352    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.064544 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.064474    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.064719 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.064698    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.066632 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.066604    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.073907 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.073882    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mwvz\" (UniqueName: \"kubernetes.io/projected/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-api-access-6mwvz\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.164836 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.164740    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-wtmp\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.164836 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.164784    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-sys\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.164836 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.164828    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-textfile\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165092 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.164853    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-accelerators-collector-config\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165092 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.164874    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xrg\" (UniqueName: \"kubernetes.io/projected/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-kube-api-access-b4xrg\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165092 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.164877    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-sys\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165092 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.164890    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-root\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165092 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.164942    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-wtmp\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165092 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.164976    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165092 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.164995    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-root\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165423 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.165105    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-tls\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165423 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.165134    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-metrics-client-ca\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165423 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.165219    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-textfile\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165423 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:43.165246    2550 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 06 20:36:43.165423 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:43.165302    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-tls podName:bcc61e35-edac-4bcf-a81a-80dd8ea0a029 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:43.665285754 +0000 UTC m=+116.606589599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-tls") pod "node-exporter-sldgn" (UID: "bcc61e35-edac-4bcf-a81a-80dd8ea0a029") : secret "node-exporter-tls" not found
Jul 06 20:36:43.165637 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.165505    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-accelerators-collector-config\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.165637 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.165553    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-metrics-client-ca\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.167521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.167497    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.173841 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.173823    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xrg\" (UniqueName: \"kubernetes.io/projected/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-kube-api-access-b4xrg\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.468551 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.468459    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69c4ee10-5ebb-44d8-9552-da4996f18848-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:43.471257 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.471227    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69c4ee10-5ebb-44d8-9552-da4996f18848-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-45gjw\" (UID: \"69c4ee10-5ebb-44d8-9552-da4996f18848\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:43.569508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.569474    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:43.569675 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:43.569648    2550 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 06 20:36:43.569739 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:43.569723    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-tls podName:343a6f78-6839-47a4-9a6c-4f71f52aa9f0 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:44.569704255 +0000 UTC m=+117.511008085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-v2h4j" (UID: "343a6f78-6839-47a4-9a6c-4f71f52aa9f0") : secret "kube-state-metrics-tls" not found
Jul 06 20:36:43.670511 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.670469    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-tls\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.673380 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.673359    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcc61e35-edac-4bcf-a81a-80dd8ea0a029-node-exporter-tls\") pod \"node-exporter-sldgn\" (UID: \"bcc61e35-edac-4bcf-a81a-80dd8ea0a029\") " pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.754295 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.754215    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"
Jul 06 20:36:43.820992 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.820960    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sldgn"
Jul 06 20:36:43.891629 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.891601    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:36:43.895435 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.895410    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.898188 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.898158    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 06 20:36:43.898188 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.898172    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 06 20:36:43.898382 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.898225    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 06 20:36:43.898907 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.898886    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 06 20:36:43.899032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.899002    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 06 20:36:43.899106 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.899033    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 06 20:36:43.899961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.899932    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 06 20:36:43.901321 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.901175    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 06 20:36:43.901681 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.901646    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6r66r\""
Jul 06 20:36:43.908649 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.908629    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 06 20:36:43.915095 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.915077    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:36:43.973995 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.973965    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974178 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974018    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974178 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974055    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-out\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974178 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974128    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974348 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974224    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs597\" (UniqueName: \"kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-kube-api-access-hs597\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974348 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974257    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974348 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974285    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974348 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974308    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974618 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974352    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-volume\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974618 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974377    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974618 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974413    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-web-config\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974618 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974443    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:43.974618 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:43.974510    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076135 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076100    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs597\" (UniqueName: \"kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-kube-api-access-hs597\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076135 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076136    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076378 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076159    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076378 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076182    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076378 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076222    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-volume\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076378 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076245    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076378 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076281    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-web-config\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076378 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076331    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076719 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076374    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076719 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076421    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076719 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076461    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076719 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076488    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-out\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.076719 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.076527    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.077259 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.077227    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.077391 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:44.077366    2550 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 06 20:36:44.077447 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:36:44.077423    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-main-tls podName:1735249d-0305-45bd-aefb-3dd636ab9d4e nodeName:}" failed. No retries permitted until 2026-07-06 20:36:44.57740561 +0000 UTC m=+117.518709662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e") : secret "alertmanager-main-tls" not found
Jul 06 20:36:44.077512 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.077471    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.080371 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.080297    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-web-config\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.081111 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.080797    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.081111 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.081058    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.081111 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.081073    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.081314 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.081234    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.082941 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.081703    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-volume\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.082941 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.082219    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.086242 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.085699    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs597\" (UniqueName: \"kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-kube-api-access-hs597\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.088934 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.088911    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-out\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.091636 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.091612    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.581307 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.581274    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.581479 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.581417    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:44.584017 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.583994    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.584137 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.584023    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/343a6f78-6839-47a4-9a6c-4f71f52aa9f0-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-v2h4j\" (UID: \"343a6f78-6839-47a4-9a6c-4f71f52aa9f0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:44.714333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.714294    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"
Jul 06 20:36:44.809139 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.809110    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:36:44.887904 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:44.887828    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9cb44d9bd-8hwkc"]
Jul 06 20:36:46.450949 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:46.450912    2550 patch_prober.go:28] interesting pod/image-registry-7558884f8-xtl65 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 06 20:36:46.451374 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:46.450986    2550 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7558884f8-xtl65" podUID="354a41da-1c7d-4c43-a17c-bcea3ede2a96" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:36:47.907782 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:47.907751    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:36:49.057832 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.057782    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:36:49.160581 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.160420    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:36:49.160762 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.160604    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.164028 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.163868    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s2tml\""
Jul 06 20:36:49.164028 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.163932    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 06 20:36:49.164211 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.164121    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 06 20:36:49.164211 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.164152    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 06 20:36:49.164314 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.164298    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 06 20:36:49.164372 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.164352    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 06 20:36:49.164651 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.164471    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 06 20:36:49.164740 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.164675    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 06 20:36:49.164888 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.164873    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 06 20:36:49.166065 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.165051    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 06 20:36:49.166065 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.165227    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3bj5u64ukmqrl\""
Jul 06 20:36:49.166065 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.165349    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 06 20:36:49.166065 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.165547    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 06 20:36:49.166065 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.165616    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 06 20:36:49.166065 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.165860    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 06 20:36:49.166802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.166709    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 06 20:36:49.167608 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.167586    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 06 20:36:49.221424 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221390    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221566 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221454    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221566 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221480    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-web-config\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221566 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221530    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221715 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221581    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv9vl\" (UniqueName: \"kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-kube-api-access-dv9vl\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221715 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221629    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221715 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221672    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-config-out\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221715 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221701    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221913 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221724    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221913 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221758    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221913 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221786    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-config\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221913 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221831    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.221913 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221855    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.222143 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221914    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.222143 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221942    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.222143 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221968    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.222143 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.221992    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.222143 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.222061    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.222143 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.222097    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.222380 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.222160    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.322571 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322540    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.322750 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322584    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-config-out\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.322750 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322611    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.322750 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322631    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.322750 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322663    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.322750 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322688    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-config\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.322750 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322708    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.322750 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322729    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.323123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322768    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.323123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322797    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.323123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322838    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.323123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322863    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.323123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322899    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.323123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322924    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.323123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322956    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.323123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.322988    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.323497 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.323390    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.323497 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.323402    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.325150 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.324401    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.325150 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.324436    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-web-config\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.325150 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.324463    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.325150 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.324499    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv9vl\" (UniqueName: \"kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-kube-api-access-dv9vl\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.325150 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.324589    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.325150 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.325080    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.325868 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.325830    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.326281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.326258    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.327232 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.327164    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.328176 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.327802    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.329120 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.328913    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.329335 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.329228    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-config\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.330050 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.329343    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.330050 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.329823    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.330050 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.329887    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-config-out\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.330050 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.329921    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.330297 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.330249    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.330472 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.330451    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-web-config\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.330547 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.330453    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.330971 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.330945    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.331638 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.331613    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.333041 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.333023    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv9vl\" (UniqueName: \"kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-kube-api-access-dv9vl\") pod \"prometheus-k8s-0\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.473821 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.473777    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:36:49.905424 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:49.905399    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw"]
Jul 06 20:36:49.908684 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:49.908642    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c4ee10_5ebb_44d8_9552_da4996f18848.slice/crio-49dae8b513a3ab6b673b3d8d55c97b89a515e8906277010857d8cc1d07c933c9 WatchSource:0}: Error finding container 49dae8b513a3ab6b673b3d8d55c97b89a515e8906277010857d8cc1d07c933c9: Status 404 returned error can't find the container with id 49dae8b513a3ab6b673b3d8d55c97b89a515e8906277010857d8cc1d07c933c9
Jul 06 20:36:50.031642 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.031598    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-qkh89" event={"ID":"6c4dcd10-1f2f-4405-805a-9694a2122c6f","Type":"ContainerStarted","Data":"a00baa3e4e39ac762baee5a0af6c4009d9669a495093db78396d53b062a6cb45"}
Jul 06 20:36:50.031789 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.031746    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-qkh89"
Jul 06 20:36:50.033431 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.033398    2550 patch_prober.go:28] interesting pod/downloads-685bfb5f64-qkh89 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.13:8080/\": dial tcp 10.132.0.13:8080: connect: connection refused" start-of-body=
Jul 06 20:36:50.033521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.033476    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-qkh89" podUID="6c4dcd10-1f2f-4405-805a-9694a2122c6f" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.13:8080/\": dial tcp 10.132.0.13:8080: connect: connection refused"
Jul 06 20:36:50.034135 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.034108    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ct7m" event={"ID":"1f3b2b13-dcaa-4ce6-8e00-201340f91ed0","Type":"ContainerStarted","Data":"a293aa47ad7f4789222f490c5502e1be51751ba1780d74eda9dafad409c6e79f"}
Jul 06 20:36:50.035308 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.035286    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sldgn" event={"ID":"bcc61e35-edac-4bcf-a81a-80dd8ea0a029","Type":"ContainerStarted","Data":"20614b57c34b251ecc9a1b4ddf8a4d7c81fe3f4f9886c7bd4766949153e20cde"}
Jul 06 20:36:50.036684 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.036665    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw" event={"ID":"69c4ee10-5ebb-44d8-9552-da4996f18848","Type":"ContainerStarted","Data":"f415e72b2df171a9c3eeedec424e02e37eed2e453dcb1dfd3fcbfd7bdc028cb6"}
Jul 06 20:36:50.036787 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.036702    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw" event={"ID":"69c4ee10-5ebb-44d8-9552-da4996f18848","Type":"ContainerStarted","Data":"49dae8b513a3ab6b673b3d8d55c97b89a515e8906277010857d8cc1d07c933c9"}
Jul 06 20:36:50.049937 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.049897    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-qkh89" podStartSLOduration=2.223377501 podStartE2EDuration="22.04988486s" totalImagesPullingTime="19.826507359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:50.049233606 +0000 UTC m=+122.990537466" watchObservedRunningTime="2026-07-06 20:36:50.04988486 +0000 UTC m=+122.991188706"
Jul 06 20:36:50.077577 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.077535    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4ct7m" podStartSLOduration=33.103588807 podStartE2EDuration="51.077522503s" totalImagesPullingTime="17.973933696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:50.076017901 +0000 UTC m=+123.017321747" watchObservedRunningTime="2026-07-06 20:36:50.077522503 +0000 UTC m=+123.018826347"
Jul 06 20:36:50.128332 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.128310    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j"]
Jul 06 20:36:50.130122 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:50.130099    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343a6f78_6839_47a4_9a6c_4f71f52aa9f0.slice/crio-463af94a92f6008b2d5067ae83c77518b86f117d4ea5742e7a5cb0f8069c4640 WatchSource:0}: Error finding container 463af94a92f6008b2d5067ae83c77518b86f117d4ea5742e7a5cb0f8069c4640: Status 404 returned error can't find the container with id 463af94a92f6008b2d5067ae83c77518b86f117d4ea5742e7a5cb0f8069c4640
Jul 06 20:36:50.135866 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.135842    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:36:50.137586 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:50.137564    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1735249d_0305_45bd_aefb_3dd636ab9d4e.slice/crio-68793fd64da574d398a5d3cec6c4ede3122093ed0862532bb325493c602bd988 WatchSource:0}: Error finding container 68793fd64da574d398a5d3cec6c4ede3122093ed0862532bb325493c602bd988: Status 404 returned error can't find the container with id 68793fd64da574d398a5d3cec6c4ede3122093ed0862532bb325493c602bd988
Jul 06 20:36:50.145609 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.145589    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:36:50.148163 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:50.147913    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003d8026_ce57_4ee3_81b9_e467ff0d590f.slice/crio-2819141dcbcc541a09220bfb9b04763f2543badf49e277a0315cbc1c6a814244 WatchSource:0}: Error finding container 2819141dcbcc541a09220bfb9b04763f2543badf49e277a0315cbc1c6a814244: Status 404 returned error can't find the container with id 2819141dcbcc541a09220bfb9b04763f2543badf49e277a0315cbc1c6a814244
Jul 06 20:36:50.815465 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:50.815427    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7558884f8-xtl65"]
Jul 06 20:36:51.042802 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:51.042756    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerStarted","Data":"2819141dcbcc541a09220bfb9b04763f2543badf49e277a0315cbc1c6a814244"}
Jul 06 20:36:51.044333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:51.044264    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j" event={"ID":"343a6f78-6839-47a4-9a6c-4f71f52aa9f0","Type":"ContainerStarted","Data":"463af94a92f6008b2d5067ae83c77518b86f117d4ea5742e7a5cb0f8069c4640"}
Jul 06 20:36:51.046484 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:51.046453    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw" event={"ID":"69c4ee10-5ebb-44d8-9552-da4996f18848","Type":"ContainerStarted","Data":"8fed29267a0b048391e691e5070f74dddd4ed1de472d070057ce128e2931d1ca"}
Jul 06 20:36:51.047871 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:51.047843    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerStarted","Data":"68793fd64da574d398a5d3cec6c4ede3122093ed0862532bb325493c602bd988"}
Jul 06 20:36:51.062367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:51.062330    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-qkh89"
Jul 06 20:36:52.058161 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:52.058121    2550 generic.go:356] "Generic (PLEG): container finished" podID="bcc61e35-edac-4bcf-a81a-80dd8ea0a029" containerID="869e8ae26320b6aa97ebd878415f62b8160493d012f0b868e31beb149ca6dec5" exitCode=0
Jul 06 20:36:52.058954 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:52.058859    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sldgn" event={"ID":"bcc61e35-edac-4bcf-a81a-80dd8ea0a029","Type":"ContainerDied","Data":"869e8ae26320b6aa97ebd878415f62b8160493d012f0b868e31beb149ca6dec5"}
Jul 06 20:36:54.075164 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:54.075005    2550 generic.go:356] "Generic (PLEG): container finished" podID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerID="2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4" exitCode=0
Jul 06 20:36:54.075164 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:54.075094    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerDied","Data":"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"}
Jul 06 20:36:54.078281 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:54.078237    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j" event={"ID":"343a6f78-6839-47a4-9a6c-4f71f52aa9f0","Type":"ContainerStarted","Data":"a6643707cb386efed6d0981280791b4239d7e9b1fadec1cac580c3d0d6fd3391"}
Jul 06 20:36:54.084659 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:54.084614    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw" event={"ID":"69c4ee10-5ebb-44d8-9552-da4996f18848","Type":"ContainerStarted","Data":"2ccd9075eb598e03f00e888a22b2174bd4899cc0693a1d214307d6b7b29cedf7"}
Jul 06 20:36:54.088332 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:54.088283    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerStarted","Data":"4bb2a44c22a579f710c2a2e8eae8b90740a7f69467c2d2e88fe2673bf7c7f42f"}
Jul 06 20:36:54.093209 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:54.093123    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sldgn" event={"ID":"bcc61e35-edac-4bcf-a81a-80dd8ea0a029","Type":"ContainerStarted","Data":"0901a097ce8b45203672126391fd1027a0d642db9966642fb20fdd89d31ac6c2"}
Jul 06 20:36:54.093209 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:54.093175    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sldgn" event={"ID":"bcc61e35-edac-4bcf-a81a-80dd8ea0a029","Type":"ContainerStarted","Data":"714d47108a00164a40779163ea6e02424248f7a6cc6b3a2dcbd098656869e258"}
Jul 06 20:36:54.165224 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:54.163748    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-45gjw" podStartSLOduration=8.528495666 podStartE2EDuration="12.163725894s" totalImagesPullingTime="3.635230228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:54.162403803 +0000 UTC m=+127.103707658" watchObservedRunningTime="2026-07-06 20:36:54.163725894 +0000 UTC m=+127.105029741"
Jul 06 20:36:55.103952 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:55.103142    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j" event={"ID":"343a6f78-6839-47a4-9a6c-4f71f52aa9f0","Type":"ContainerStarted","Data":"f8f6634a9151ebbbb11ee18f06eccd71e0a5f8169d0718194959e7937adfc345"}
Jul 06 20:36:55.103952 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:55.103253    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j" event={"ID":"343a6f78-6839-47a4-9a6c-4f71f52aa9f0","Type":"ContainerStarted","Data":"e56cf0d755ee230c9db94e90f20faa46e5e663859dbc001d8232d5167a1b947c"}
Jul 06 20:36:55.106293 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:55.106231    2550 generic.go:356] "Generic (PLEG): container finished" podID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerID="4bb2a44c22a579f710c2a2e8eae8b90740a7f69467c2d2e88fe2673bf7c7f42f" exitCode=0
Jul 06 20:36:55.106989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:55.106968    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerDied","Data":"4bb2a44c22a579f710c2a2e8eae8b90740a7f69467c2d2e88fe2673bf7c7f42f"}
Jul 06 20:36:55.129450 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:55.128949    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sldgn" podStartSLOduration=11.612729804 podStartE2EDuration="13.128922968s" totalImagesPullingTime="1.395626701s" totalInitContainerRuntime="120.566463ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:54.183828776 +0000 UTC m=+127.125132625" watchObservedRunningTime="2026-07-06 20:36:55.128922968 +0000 UTC m=+128.070226816"
Jul 06 20:36:55.130279 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:55.129846    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-v2h4j" podStartSLOduration=9.494629654 podStartE2EDuration="13.129830338s" totalImagesPullingTime="3.635200684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:55.12808622 +0000 UTC m=+128.069390080" watchObservedRunningTime="2026-07-06 20:36:55.129830338 +0000 UTC m=+128.071134177"
Jul 06 20:36:57.126022 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:57.125781    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerStarted","Data":"06c603ad516b799505a49e44546c2c2fe62c53e23ba08d09430a444f557e53e8"}
Jul 06 20:36:57.126022 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:57.125899    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerStarted","Data":"e63ef78cbf589f8feedaebbd3e1c9cb4d09639a1273e43878cd5c51248cca5a9"}
Jul 06 20:36:57.213904 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:57.213350    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:36:57.216561 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:57.216530    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9866e3e7-f454-42b7-add8-3bcf2b9f797a-metrics-certs\") pod \"network-metrics-daemon-6vs4n\" (UID: \"9866e3e7-f454-42b7-add8-3bcf2b9f797a\") " pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:36:57.382913 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:57.382829    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl572\""
Jul 06 20:36:57.392268 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:57.392222    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vs4n"
Jul 06 20:36:58.505994 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:58.505970    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6vs4n"]
Jul 06 20:36:58.507964 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:36:58.507933    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9866e3e7_f454_42b7_add8_3bcf2b9f797a.slice/crio-e0bf9a91d47687e1b95316b0cd3874288666d145ad4a4f827359cce91ef20d95 WatchSource:0}: Error finding container e0bf9a91d47687e1b95316b0cd3874288666d145ad4a4f827359cce91ef20d95: Status 404 returned error can't find the container with id e0bf9a91d47687e1b95316b0cd3874288666d145ad4a4f827359cce91ef20d95
Jul 06 20:36:59.140333 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:59.140250    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerStarted","Data":"c5dd6d79f674e194b44f3cb2bba38371f36c97bfc32cd47e21758e7ddb1e9bc4"}
Jul 06 20:36:59.140490 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:59.140336    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerStarted","Data":"9d8d2c0c57d7c36371a087d30513252fea4ffe40de7ce73894504d6f7cafa801"}
Jul 06 20:36:59.140490 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:59.140385    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerStarted","Data":"80a97156491962f824f0db1293f64a0942b28a6fa7e02b0ddb94a9c7f6e8efc9"}
Jul 06 20:36:59.141693 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:59.141664    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6vs4n" event={"ID":"9866e3e7-f454-42b7-add8-3bcf2b9f797a","Type":"ContainerStarted","Data":"e0bf9a91d47687e1b95316b0cd3874288666d145ad4a4f827359cce91ef20d95"}
Jul 06 20:36:59.144574 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:59.144548    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerStarted","Data":"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"}
Jul 06 20:36:59.144695 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:36:59.144616    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerStarted","Data":"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"}
Jul 06 20:37:01.158777 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:01.158740    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerStarted","Data":"c900ddd226a6290701e3c06366c6b9bd56a64f28dd188aa5f8d819817697e646"}
Jul 06 20:37:01.160634 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:01.160606    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6vs4n" event={"ID":"9866e3e7-f454-42b7-add8-3bcf2b9f797a","Type":"ContainerStarted","Data":"ba328a828236e5ef4bbe877157314bac2f6b60b15c0f43d2fee8c3a4d961e89f"}
Jul 06 20:37:01.160766 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:01.160658    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6vs4n" event={"ID":"9866e3e7-f454-42b7-add8-3bcf2b9f797a","Type":"ContainerStarted","Data":"2627dba8a61735bafa05cab4060103119eaaa0681843391be01804300483ff37"}
Jul 06 20:37:01.194187 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:01.194139    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.755586915 podStartE2EDuration="18.194122201s" totalImagesPullingTime="7.405713419s" totalInitContainerRuntime="32.821867ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:01.191720158 +0000 UTC m=+134.133024007" watchObservedRunningTime="2026-07-06 20:37:01.194122201 +0000 UTC m=+134.135426051"
Jul 06 20:37:01.211063 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:01.210998    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6vs4n" podStartSLOduration=131.396069862 podStartE2EDuration="2m13.210975265s" totalImagesPullingTime="1.814905403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:01.209307399 +0000 UTC m=+134.150611246" watchObservedRunningTime="2026-07-06 20:37:01.210975265 +0000 UTC m=+134.152279125"
Jul 06 20:37:02.168078 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:02.168045    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerStarted","Data":"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"}
Jul 06 20:37:02.168451 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:02.168110    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerStarted","Data":"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"}
Jul 06 20:37:02.168451 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:02.168175    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerStarted","Data":"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"}
Jul 06 20:37:02.168451 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:02.168206    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerStarted","Data":"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"}
Jul 06 20:37:02.203501 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:02.203413    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.238035141 podStartE2EDuration="13.203394544s" totalImagesPullingTime="10.928245119s" totalInitContainerRuntime="37.114284ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:02.200949599 +0000 UTC m=+135.142253446" watchObservedRunningTime="2026-07-06 20:37:02.203394544 +0000 UTC m=+135.144698391"
Jul 06 20:37:04.474409 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:04.474372    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:37:09.910432 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:09.910395    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-9cb44d9bd-8hwkc" podUID="a4af1ffc-9cdb-46cc-a339-531fe2c544c5" containerName="console" containerID="cri-o://08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233" gracePeriod=15
Jul 06 20:37:10.198107 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.198082    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:37:10.200304 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.200277    2550 generic.go:356] "Generic (PLEG): container finished" podID="a4af1ffc-9cdb-46cc-a339-531fe2c544c5" containerID="08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233" exitCode=0
Jul 06 20:37:10.200429 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.200317    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cb44d9bd-8hwkc"
Jul 06 20:37:10.200429 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.200321    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cb44d9bd-8hwkc" event={"ID":"a4af1ffc-9cdb-46cc-a339-531fe2c544c5","Type":"ContainerDied","Data":"08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233"}
Jul 06 20:37:10.200429 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.200385    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cb44d9bd-8hwkc" event={"ID":"a4af1ffc-9cdb-46cc-a339-531fe2c544c5","Type":"ContainerDied","Data":"2176923295de78b92d63d31de676bef975b08fc82a698cfb169e9b4176d250a5"}
Jul 06 20:37:10.200429 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.200407    2550 scope.go:117] "RemoveContainer" containerID="08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233"
Jul 06 20:37:10.208162 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.208144    2550 scope.go:117] "RemoveContainer" containerID="08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233"
Jul 06 20:37:10.208407 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:37:10.208385    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233\": container with ID starting with 08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233 not found: ID does not exist" containerID="08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233"
Jul 06 20:37:10.208477 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.208417    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233"} err="failed to get container status \"08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233\": rpc error: code = NotFound desc = could not find container \"08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233\": container with ID starting with 08c71c2213899ed9c3791750d42c0b2ebdf54f92377c641cdff40736f74be233 not found: ID does not exist"
Jul 06 20:37:10.332447 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.332423    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-serving-cert\") pod \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") "
Jul 06 20:37:10.332626 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.332458    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-oauth-serving-cert\") pod \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") "
Jul 06 20:37:10.332626 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.332477    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-oauth-config\") pod \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") "
Jul 06 20:37:10.332626 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.332543    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd4tz\" (UniqueName: \"kubernetes.io/projected/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-kube-api-access-nd4tz\") pod \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") "
Jul 06 20:37:10.332626 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.332565    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-service-ca\") pod \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") "
Jul 06 20:37:10.332626 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.332600    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-config\") pod \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\" (UID: \"a4af1ffc-9cdb-46cc-a339-531fe2c544c5\") "
Jul 06 20:37:10.333112 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.333059    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-config" (OuterVolumeSpecName: "console-config") pod "a4af1ffc-9cdb-46cc-a339-531fe2c544c5" (UID: "a4af1ffc-9cdb-46cc-a339-531fe2c544c5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:37:10.333112 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.333068    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-service-ca" (OuterVolumeSpecName: "service-ca") pod "a4af1ffc-9cdb-46cc-a339-531fe2c544c5" (UID: "a4af1ffc-9cdb-46cc-a339-531fe2c544c5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:37:10.333112 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.333091    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a4af1ffc-9cdb-46cc-a339-531fe2c544c5" (UID: "a4af1ffc-9cdb-46cc-a339-531fe2c544c5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:37:10.334844 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.334785    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-kube-api-access-nd4tz" (OuterVolumeSpecName: "kube-api-access-nd4tz") pod "a4af1ffc-9cdb-46cc-a339-531fe2c544c5" (UID: "a4af1ffc-9cdb-46cc-a339-531fe2c544c5"). InnerVolumeSpecName "kube-api-access-nd4tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:37:10.334844 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.334835    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a4af1ffc-9cdb-46cc-a339-531fe2c544c5" (UID: "a4af1ffc-9cdb-46cc-a339-531fe2c544c5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:37:10.334982 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.334916    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a4af1ffc-9cdb-46cc-a339-531fe2c544c5" (UID: "a4af1ffc-9cdb-46cc-a339-531fe2c544c5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:37:10.433498 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.433436    2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-serving-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:10.433498 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.433461    2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-oauth-serving-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:10.433498 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.433470    2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-oauth-config\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:10.433498 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.433479    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd4tz\" (UniqueName: \"kubernetes.io/projected/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-kube-api-access-nd4tz\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:10.433498 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.433488    2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:10.433498 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.433496    2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4af1ffc-9cdb-46cc-a339-531fe2c544c5-console-config\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:10.521966 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.521944    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9cb44d9bd-8hwkc"]
Jul 06 20:37:10.524851 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:10.524830    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9cb44d9bd-8hwkc"]
Jul 06 20:37:11.478920 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:11.478882    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4af1ffc-9cdb-46cc-a339-531fe2c544c5" path="/var/lib/kubelet/pods/a4af1ffc-9cdb-46cc-a339-531fe2c544c5/volumes"
Jul 06 20:37:15.839958 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:15.839913    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7558884f8-xtl65" podUID="354a41da-1c7d-4c43-a17c-bcea3ede2a96" containerName="registry" containerID="cri-o://00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26" gracePeriod=30
Jul 06 20:37:16.117131 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.117078    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:37:16.227237 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.227197    2550 generic.go:356] "Generic (PLEG): container finished" podID="354a41da-1c7d-4c43-a17c-bcea3ede2a96" containerID="00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26" exitCode=0
Jul 06 20:37:16.227382 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.227252    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7558884f8-xtl65"
Jul 06 20:37:16.227382 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.227266    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7558884f8-xtl65" event={"ID":"354a41da-1c7d-4c43-a17c-bcea3ede2a96","Type":"ContainerDied","Data":"00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26"}
Jul 06 20:37:16.227382 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.227352    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7558884f8-xtl65" event={"ID":"354a41da-1c7d-4c43-a17c-bcea3ede2a96","Type":"ContainerDied","Data":"ac3079de51e8c1f41797d68f2dc570eb684d4f089e61f656728427c7794a05d7"}
Jul 06 20:37:16.227382 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.227379    2550 scope.go:117] "RemoveContainer" containerID="00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26"
Jul 06 20:37:16.235064 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.235047    2550 scope.go:117] "RemoveContainer" containerID="00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26"
Jul 06 20:37:16.235311 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:37:16.235293    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26\": container with ID starting with 00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26 not found: ID does not exist" containerID="00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26"
Jul 06 20:37:16.235369 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.235324    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26"} err="failed to get container status \"00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26\": rpc error: code = NotFound desc = could not find container \"00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26\": container with ID starting with 00a4bda74f12139949b708a6274f8e6baf04648e116fbc1c781d85a71e39cd26 not found: ID does not exist"
Jul 06 20:37:16.288778 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.288752    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-image-registry-private-configuration\") pod \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") "
Jul 06 20:37:16.288924 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.288783    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqgf5\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-kube-api-access-fqgf5\") pod \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") "
Jul 06 20:37:16.288924 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.288826    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-trusted-ca\") pod \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") "
Jul 06 20:37:16.289030 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.288961    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-installation-pull-secrets\") pod \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") "
Jul 06 20:37:16.289030 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.289009    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-bound-sa-token\") pod \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") "
Jul 06 20:37:16.289129 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.289108    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") pod \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") "
Jul 06 20:37:16.289187 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.289139    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-certificates\") pod \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") "
Jul 06 20:37:16.289187 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.289169    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/354a41da-1c7d-4c43-a17c-bcea3ede2a96-ca-trust-extracted\") pod \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\" (UID: \"354a41da-1c7d-4c43-a17c-bcea3ede2a96\") "
Jul 06 20:37:16.289285 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.289231    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "354a41da-1c7d-4c43-a17c-bcea3ede2a96" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:37:16.290090 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.289517    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-trusted-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:16.290090 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.289995    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "354a41da-1c7d-4c43-a17c-bcea3ede2a96" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:37:16.291423 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.291392    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "354a41da-1c7d-4c43-a17c-bcea3ede2a96" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:37:16.291514 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.291462    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "354a41da-1c7d-4c43-a17c-bcea3ede2a96" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:37:16.291563 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.291519    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-kube-api-access-fqgf5" (OuterVolumeSpecName: "kube-api-access-fqgf5") pod "354a41da-1c7d-4c43-a17c-bcea3ede2a96" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96"). InnerVolumeSpecName "kube-api-access-fqgf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:37:16.291612 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.291586    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "354a41da-1c7d-4c43-a17c-bcea3ede2a96" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:37:16.291840 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.291803    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "354a41da-1c7d-4c43-a17c-bcea3ede2a96" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:37:16.300094 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.300071    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354a41da-1c7d-4c43-a17c-bcea3ede2a96-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "354a41da-1c7d-4c43-a17c-bcea3ede2a96" (UID: "354a41da-1c7d-4c43-a17c-bcea3ede2a96"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:37:16.390733 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.390673    2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-tls\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:16.390733 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.390695    2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/354a41da-1c7d-4c43-a17c-bcea3ede2a96-registry-certificates\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:16.390733 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.390705    2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/354a41da-1c7d-4c43-a17c-bcea3ede2a96-ca-trust-extracted\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:16.390733 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.390715    2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-image-registry-private-configuration\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:16.390733 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.390724    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqgf5\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-kube-api-access-fqgf5\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:16.390733 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.390732    2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/354a41da-1c7d-4c43-a17c-bcea3ede2a96-installation-pull-secrets\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:16.391012 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.390740    2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/354a41da-1c7d-4c43-a17c-bcea3ede2a96-bound-sa-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:37:16.548904 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.548877    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7558884f8-xtl65"]
Jul 06 20:37:16.550992 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:16.550971    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7558884f8-xtl65"]
Jul 06 20:37:17.474432 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:17.474396    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354a41da-1c7d-4c43-a17c-bcea3ede2a96" path="/var/lib/kubelet/pods/354a41da-1c7d-4c43-a17c-bcea3ede2a96/volumes"
Jul 06 20:37:21.248648 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:21.248615    2550 generic.go:356] "Generic (PLEG): container finished" podID="eaf12a17-0dd7-4e9d-939f-33442530321f" containerID="954a79d2b2ab7f0267b21c82ee17e743eeb8b6b9adb3bfead269948ab741fb90" exitCode=0
Jul 06 20:37:21.249054 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:21.248687    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8" event={"ID":"eaf12a17-0dd7-4e9d-939f-33442530321f","Type":"ContainerDied","Data":"954a79d2b2ab7f0267b21c82ee17e743eeb8b6b9adb3bfead269948ab741fb90"}
Jul 06 20:37:21.249054 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:21.249039    2550 scope.go:117] "RemoveContainer" containerID="954a79d2b2ab7f0267b21c82ee17e743eeb8b6b9adb3bfead269948ab741fb90"
Jul 06 20:37:22.254407 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:22.254376    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8f2m8" event={"ID":"eaf12a17-0dd7-4e9d-939f-33442530321f","Type":"ContainerStarted","Data":"e863cb365b674e5e800f933d8ddfc0c3c3a85473935a31bfd7036833716f5415"}
Jul 06 20:37:49.474703 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:49.474669    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:37:49.492400 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:49.492375    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:37:50.383260 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:37:50.383235    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:03.191074 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.191041    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:38:03.191552 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.191328    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="alertmanager" containerID="cri-o://e63ef78cbf589f8feedaebbd3e1c9cb4d09639a1273e43878cd5c51248cca5a9" gracePeriod=120
Jul 06 20:38:03.191552 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.191377    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy-metric" containerID="cri-o://c5dd6d79f674e194b44f3cb2bba38371f36c97bfc32cd47e21758e7ddb1e9bc4" gracePeriod=120
Jul 06 20:38:03.191552 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.191411    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="prom-label-proxy" containerID="cri-o://c900ddd226a6290701e3c06366c6b9bd56a64f28dd188aa5f8d819817697e646" gracePeriod=120
Jul 06 20:38:03.191552 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.191456    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="config-reloader" containerID="cri-o://06c603ad516b799505a49e44546c2c2fe62c53e23ba08d09430a444f557e53e8" gracePeriod=120
Jul 06 20:38:03.191552 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.191466    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy-web" containerID="cri-o://80a97156491962f824f0db1293f64a0942b28a6fa7e02b0ddb94a9c7f6e8efc9" gracePeriod=120
Jul 06 20:38:03.191835 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.191481    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy" containerID="cri-o://9d8d2c0c57d7c36371a087d30513252fea4ffe40de7ce73894504d6f7cafa801" gracePeriod=120
Jul 06 20:38:03.420264 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.420234    2550 generic.go:356] "Generic (PLEG): container finished" podID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerID="c900ddd226a6290701e3c06366c6b9bd56a64f28dd188aa5f8d819817697e646" exitCode=0
Jul 06 20:38:03.420264 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.420259    2550 generic.go:356] "Generic (PLEG): container finished" podID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerID="9d8d2c0c57d7c36371a087d30513252fea4ffe40de7ce73894504d6f7cafa801" exitCode=0
Jul 06 20:38:03.420264 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.420267    2550 generic.go:356] "Generic (PLEG): container finished" podID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerID="06c603ad516b799505a49e44546c2c2fe62c53e23ba08d09430a444f557e53e8" exitCode=0
Jul 06 20:38:03.420444 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.420274    2550 generic.go:356] "Generic (PLEG): container finished" podID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerID="e63ef78cbf589f8feedaebbd3e1c9cb4d09639a1273e43878cd5c51248cca5a9" exitCode=0
Jul 06 20:38:03.420444 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.420303    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerDied","Data":"c900ddd226a6290701e3c06366c6b9bd56a64f28dd188aa5f8d819817697e646"}
Jul 06 20:38:03.420444 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.420364    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerDied","Data":"9d8d2c0c57d7c36371a087d30513252fea4ffe40de7ce73894504d6f7cafa801"}
Jul 06 20:38:03.420444 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.420389    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerDied","Data":"06c603ad516b799505a49e44546c2c2fe62c53e23ba08d09430a444f557e53e8"}
Jul 06 20:38:03.420444 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:03.420413    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerDied","Data":"e63ef78cbf589f8feedaebbd3e1c9cb4d09639a1273e43878cd5c51248cca5a9"}
Jul 06 20:38:04.429683 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.429653    2550 generic.go:356] "Generic (PLEG): container finished" podID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerID="c5dd6d79f674e194b44f3cb2bba38371f36c97bfc32cd47e21758e7ddb1e9bc4" exitCode=0
Jul 06 20:38:04.429683 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.429677    2550 generic.go:356] "Generic (PLEG): container finished" podID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerID="80a97156491962f824f0db1293f64a0942b28a6fa7e02b0ddb94a9c7f6e8efc9" exitCode=0
Jul 06 20:38:04.430179 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.429724    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerDied","Data":"c5dd6d79f674e194b44f3cb2bba38371f36c97bfc32cd47e21758e7ddb1e9bc4"}
Jul 06 20:38:04.430179 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.429832    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerDied","Data":"80a97156491962f824f0db1293f64a0942b28a6fa7e02b0ddb94a9c7f6e8efc9"}
Jul 06 20:38:04.461658 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.457908    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:04.515902 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.515825    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.515902 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.515888    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-volume\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516095 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516073    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-trusted-ca-bundle\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516146 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516133    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-web\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516179 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516159    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-cluster-tls-config\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516223 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516177    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516277 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516222    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-metrics-client-ca\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516277 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516264    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-main-db\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516378 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516295    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-main-tls\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516378 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516344    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-tls-assets\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516482 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516378    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs597\" (UniqueName: \"kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-kube-api-access-hs597\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516482 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516401    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-web-config\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.516482 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516430    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-out\") pod \"1735249d-0305-45bd-aefb-3dd636ab9d4e\" (UID: \"1735249d-0305-45bd-aefb-3dd636ab9d4e\") "
Jul 06 20:38:04.517039 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516801    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:04.517039 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.516948    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:04.518745 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.518708    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:04.518745 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.518718    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:38:04.519097 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.519044    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:04.519364 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.519332    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-kube-api-access-hs597" (OuterVolumeSpecName: "kube-api-access-hs597") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "kube-api-access-hs597". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:38:04.519745 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.519714    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:38:04.519863 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.519747    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-volume" (OuterVolumeSpecName: "config-volume") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:04.520219 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.520199    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-out" (OuterVolumeSpecName: "config-out") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:38:04.520711 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.520690    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:04.521465 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.521445    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:04.526291 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.526270    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:04.531335 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.531300    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-web-config" (OuterVolumeSpecName: "web-config") pod "1735249d-0305-45bd-aefb-3dd636ab9d4e" (UID: "1735249d-0305-45bd-aefb-3dd636ab9d4e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:04.617526 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617505    2550 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-main-db\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617526 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617525    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-main-tls\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617535    2550 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-tls-assets\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617544    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs597\" (UniqueName: \"kubernetes.io/projected/1735249d-0305-45bd-aefb-3dd636ab9d4e-kube-api-access-hs597\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617553    2550 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-web-config\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617560    2550 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-out\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617569    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617577    2550 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-config-volume\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617584    2550 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617592    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617600    2550 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-cluster-tls-config\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617608    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1735249d-0305-45bd-aefb-3dd636ab9d4e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:04.617650 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:04.617617    2550 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1735249d-0305-45bd-aefb-3dd636ab9d4e-metrics-client-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:05.435829 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.435784    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1735249d-0305-45bd-aefb-3dd636ab9d4e","Type":"ContainerDied","Data":"68793fd64da574d398a5d3cec6c4ede3122093ed0862532bb325493c602bd988"}
Jul 06 20:38:05.436180 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.435874    2550 scope.go:117] "RemoveContainer" containerID="c900ddd226a6290701e3c06366c6b9bd56a64f28dd188aa5f8d819817697e646"
Jul 06 20:38:05.436180 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.435877    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.443422 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.443403    2550 scope.go:117] "RemoveContainer" containerID="c5dd6d79f674e194b44f3cb2bba38371f36c97bfc32cd47e21758e7ddb1e9bc4"
Jul 06 20:38:05.450676 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.450658    2550 scope.go:117] "RemoveContainer" containerID="9d8d2c0c57d7c36371a087d30513252fea4ffe40de7ce73894504d6f7cafa801"
Jul 06 20:38:05.456922 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.456906    2550 scope.go:117] "RemoveContainer" containerID="80a97156491962f824f0db1293f64a0942b28a6fa7e02b0ddb94a9c7f6e8efc9"
Jul 06 20:38:05.461078 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.461060    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:38:05.466017 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.465999    2550 scope.go:117] "RemoveContainer" containerID="06c603ad516b799505a49e44546c2c2fe62c53e23ba08d09430a444f557e53e8"
Jul 06 20:38:05.466077 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.466043    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:38:05.473230 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.473207    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" path="/var/lib/kubelet/pods/1735249d-0305-45bd-aefb-3dd636ab9d4e/volumes"
Jul 06 20:38:05.473761 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.473744    2550 scope.go:117] "RemoveContainer" containerID="e63ef78cbf589f8feedaebbd3e1c9cb4d09639a1273e43878cd5c51248cca5a9"
Jul 06 20:38:05.480261 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:05.480248    2550 scope.go:117] "RemoveContainer" containerID="4bb2a44c22a579f710c2a2e8eae8b90740a7f69467c2d2e88fe2673bf7c7f42f"
Jul 06 20:38:07.522946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.522913    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:38:07.523358 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.523197    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="prometheus" containerID="cri-o://80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704" gracePeriod=600
Jul 06 20:38:07.523358 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.523230    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy" containerID="cri-o://10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7" gracePeriod=600
Jul 06 20:38:07.523358 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.523262    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy-web" containerID="cri-o://22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d" gracePeriod=600
Jul 06 20:38:07.523358 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.523280    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="config-reloader" containerID="cri-o://1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50" gracePeriod=600
Jul 06 20:38:07.523358 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.523288    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1" gracePeriod=600
Jul 06 20:38:07.523358 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.523240    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="thanos-sidecar" containerID="cri-o://1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b" gracePeriod=600
Jul 06 20:38:07.768306 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.768278    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:07.843033 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.842997    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-thanos-prometheus-http-client-file\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843222 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843052    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-config\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843222 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843084    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv9vl\" (UniqueName: \"kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-kube-api-access-dv9vl\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843222 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843108    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-metrics-client-ca\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843222 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843129    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-2\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843436 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843259    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843436 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843303    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-kube-rbac-proxy\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843436 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843347    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-metrics-client-certs\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843436 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843384    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-web-config\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843436 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843415    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-trusted-ca-bundle\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843439    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-0\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843501    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-config-out\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843501    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843510    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843526    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-grpc-tls\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843550    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-1\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843584    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-tls-assets\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843609    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843640    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-db\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843692    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-kubelet-serving-ca-bundle\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.843736 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843726    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-tls\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.844335 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.843758    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-serving-certs-ca-bundle\") pod \"003d8026-ce57-4ee3-81b9-e467ff0d590f\" (UID: \"003d8026-ce57-4ee3-81b9-e467ff0d590f\") "
Jul 06 20:38:07.844335 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.844004    2550 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-metrics-client-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.844335 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.844019    2550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.844335 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.844295    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:07.844961 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.844934    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:07.845221 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.845196    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:07.846221 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.846193    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:07.846464 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.846433    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-kube-api-access-dv9vl" (OuterVolumeSpecName: "kube-api-access-dv9vl") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "kube-api-access-dv9vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:38:07.846655 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.846628    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:07.846833 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.846787    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:07.846949 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.846919    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:07.847283 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.847257    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-config" (OuterVolumeSpecName: "config") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:07.847371 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.847310    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:38:07.848085 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.848051    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:07.848525 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.848493    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-config-out" (OuterVolumeSpecName: "config-out") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:38:07.848616 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.848527    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:07.849069 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.849019    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:07.849182 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.849164    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:38:07.849376 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.849358    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:07.850334 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.850313    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:07.859161 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.859132    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-web-config" (OuterVolumeSpecName: "web-config") pod "003d8026-ce57-4ee3-81b9-e467ff0d590f" (UID: "003d8026-ce57-4ee3-81b9-e467ff0d590f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:07.945127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945092    2550 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945121    2550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945132    2550 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945142    2550 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945151    2550 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-config\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945160    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv9vl\" (UniqueName: \"kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-kube-api-access-dv9vl\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945170    2550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945180    2550 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-kube-rbac-proxy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945188    2550 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-metrics-client-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945196    2550 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-web-config\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945204    2550 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945212    2550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945219    2550 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-config-out\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945227    2550 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-grpc-tls\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945234    2550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945244    2550 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/003d8026-ce57-4ee3-81b9-e467ff0d590f-tls-assets\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945251    2550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/003d8026-ce57-4ee3-81b9-e467ff0d590f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:07.945342 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:07.945260    2550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/003d8026-ce57-4ee3-81b9-e467ff0d590f-prometheus-k8s-db\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:38:08.451192 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451153    2550 generic.go:356] "Generic (PLEG): container finished" podID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerID="35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1" exitCode=0
Jul 06 20:38:08.451192 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451185    2550 generic.go:356] "Generic (PLEG): container finished" podID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerID="10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7" exitCode=0
Jul 06 20:38:08.451192 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451192    2550 generic.go:356] "Generic (PLEG): container finished" podID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerID="22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d" exitCode=0
Jul 06 20:38:08.451192 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451199    2550 generic.go:356] "Generic (PLEG): container finished" podID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerID="1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b" exitCode=0
Jul 06 20:38:08.451437 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451206    2550 generic.go:356] "Generic (PLEG): container finished" podID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerID="1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50" exitCode=0
Jul 06 20:38:08.451437 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451211    2550 generic.go:356] "Generic (PLEG): container finished" podID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerID="80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704" exitCode=0
Jul 06 20:38:08.451437 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451240    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.451437 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451241    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerDied","Data":"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"}
Jul 06 20:38:08.451437 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451336    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerDied","Data":"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"}
Jul 06 20:38:08.451437 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451364    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerDied","Data":"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"}
Jul 06 20:38:08.451437 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451388    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerDied","Data":"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"}
Jul 06 20:38:08.451437 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451401    2550 scope.go:117] "RemoveContainer" containerID="35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"
Jul 06 20:38:08.451437 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451421    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerDied","Data":"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"}
Jul 06 20:38:08.451741 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451454    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerDied","Data":"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"}
Jul 06 20:38:08.451741 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.451481    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"003d8026-ce57-4ee3-81b9-e467ff0d590f","Type":"ContainerDied","Data":"2819141dcbcc541a09220bfb9b04763f2543badf49e277a0315cbc1c6a814244"}
Jul 06 20:38:08.459721 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.459698    2550 scope.go:117] "RemoveContainer" containerID="10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"
Jul 06 20:38:08.466243 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.466219    2550 scope.go:117] "RemoveContainer" containerID="22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"
Jul 06 20:38:08.472332 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.472317    2550 scope.go:117] "RemoveContainer" containerID="1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"
Jul 06 20:38:08.478742 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.478604    2550 scope.go:117] "RemoveContainer" containerID="1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"
Jul 06 20:38:08.480381 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.480361    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:38:08.485685 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.485657    2550 scope.go:117] "RemoveContainer" containerID="80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"
Jul 06 20:38:08.487117 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.487096    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:38:08.492460 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.492447    2550 scope.go:117] "RemoveContainer" containerID="2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"
Jul 06 20:38:08.498676 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.498658    2550 scope.go:117] "RemoveContainer" containerID="35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"
Jul 06 20:38:08.498927 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:38:08.498908    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": container with ID starting with 35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1 not found: ID does not exist" containerID="35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"
Jul 06 20:38:08.499019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.498938    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"} err="failed to get container status \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": rpc error: code = NotFound desc = could not find container \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": container with ID starting with 35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1 not found: ID does not exist"
Jul 06 20:38:08.499019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.498962    2550 scope.go:117] "RemoveContainer" containerID="10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"
Jul 06 20:38:08.499233 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:38:08.499217    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": container with ID starting with 10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7 not found: ID does not exist" containerID="10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"
Jul 06 20:38:08.499273 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.499241    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"} err="failed to get container status \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": rpc error: code = NotFound desc = could not find container \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": container with ID starting with 10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7 not found: ID does not exist"
Jul 06 20:38:08.499273 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.499258    2550 scope.go:117] "RemoveContainer" containerID="22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"
Jul 06 20:38:08.499459 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:38:08.499443    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": container with ID starting with 22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d not found: ID does not exist" containerID="22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"
Jul 06 20:38:08.499513 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.499466    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"} err="failed to get container status \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": rpc error: code = NotFound desc = could not find container \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": container with ID starting with 22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d not found: ID does not exist"
Jul 06 20:38:08.499513 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.499488    2550 scope.go:117] "RemoveContainer" containerID="1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"
Jul 06 20:38:08.499713 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:38:08.499694    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": container with ID starting with 1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b not found: ID does not exist" containerID="1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"
Jul 06 20:38:08.499763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.499716    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"} err="failed to get container status \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": rpc error: code = NotFound desc = could not find container \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": container with ID starting with 1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b not found: ID does not exist"
Jul 06 20:38:08.499763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.499727    2550 scope.go:117] "RemoveContainer" containerID="1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"
Jul 06 20:38:08.499926 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:38:08.499909    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": container with ID starting with 1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50 not found: ID does not exist" containerID="1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"
Jul 06 20:38:08.499979 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.499930    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"} err="failed to get container status \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": rpc error: code = NotFound desc = could not find container \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": container with ID starting with 1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50 not found: ID does not exist"
Jul 06 20:38:08.499979 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.499945    2550 scope.go:117] "RemoveContainer" containerID="80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"
Jul 06 20:38:08.500188 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:38:08.500173    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": container with ID starting with 80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704 not found: ID does not exist" containerID="80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"
Jul 06 20:38:08.500230 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.500191    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"} err="failed to get container status \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": rpc error: code = NotFound desc = could not find container \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": container with ID starting with 80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704 not found: ID does not exist"
Jul 06 20:38:08.500230 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.500203    2550 scope.go:117] "RemoveContainer" containerID="2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"
Jul 06 20:38:08.500406 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:38:08.500391    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": container with ID starting with 2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4 not found: ID does not exist" containerID="2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"
Jul 06 20:38:08.500447 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.500409    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"} err="failed to get container status \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": rpc error: code = NotFound desc = could not find container \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": container with ID starting with 2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4 not found: ID does not exist"
Jul 06 20:38:08.500447 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.500420    2550 scope.go:117] "RemoveContainer" containerID="35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"
Jul 06 20:38:08.500606 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.500590    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"} err="failed to get container status \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": rpc error: code = NotFound desc = could not find container \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": container with ID starting with 35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1 not found: ID does not exist"
Jul 06 20:38:08.500649 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.500606    2550 scope.go:117] "RemoveContainer" containerID="10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"
Jul 06 20:38:08.500894 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.500864    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"} err="failed to get container status \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": rpc error: code = NotFound desc = could not find container \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": container with ID starting with 10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7 not found: ID does not exist"
Jul 06 20:38:08.500894 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.500890    2550 scope.go:117] "RemoveContainer" containerID="22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"
Jul 06 20:38:08.501109 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.501093    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"} err="failed to get container status \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": rpc error: code = NotFound desc = could not find container \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": container with ID starting with 22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d not found: ID does not exist"
Jul 06 20:38:08.501158 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.501110    2550 scope.go:117] "RemoveContainer" containerID="1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"
Jul 06 20:38:08.501330 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.501310    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"} err="failed to get container status \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": rpc error: code = NotFound desc = could not find container \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": container with ID starting with 1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b not found: ID does not exist"
Jul 06 20:38:08.501378 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.501330    2550 scope.go:117] "RemoveContainer" containerID="1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"
Jul 06 20:38:08.501552 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.501536    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"} err="failed to get container status \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": rpc error: code = NotFound desc = could not find container \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": container with ID starting with 1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50 not found: ID does not exist"
Jul 06 20:38:08.501607 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.501552    2550 scope.go:117] "RemoveContainer" containerID="80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"
Jul 06 20:38:08.501782 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.501765    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"} err="failed to get container status \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": rpc error: code = NotFound desc = could not find container \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": container with ID starting with 80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704 not found: ID does not exist"
Jul 06 20:38:08.501858 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.501782    2550 scope.go:117] "RemoveContainer" containerID="2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"
Jul 06 20:38:08.502017 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.502000    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"} err="failed to get container status \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": rpc error: code = NotFound desc = could not find container \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": container with ID starting with 2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4 not found: ID does not exist"
Jul 06 20:38:08.502058 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.502017    2550 scope.go:117] "RemoveContainer" containerID="35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"
Jul 06 20:38:08.502238 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.502221    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"} err="failed to get container status \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": rpc error: code = NotFound desc = could not find container \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": container with ID starting with 35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1 not found: ID does not exist"
Jul 06 20:38:08.502285 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.502238    2550 scope.go:117] "RemoveContainer" containerID="10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"
Jul 06 20:38:08.502424 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.502410    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"} err="failed to get container status \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": rpc error: code = NotFound desc = could not find container \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": container with ID starting with 10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7 not found: ID does not exist"
Jul 06 20:38:08.502464 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.502425    2550 scope.go:117] "RemoveContainer" containerID="22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"
Jul 06 20:38:08.502627 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.502610    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"} err="failed to get container status \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": rpc error: code = NotFound desc = could not find container \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": container with ID starting with 22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d not found: ID does not exist"
Jul 06 20:38:08.502671 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.502627    2550 scope.go:117] "RemoveContainer" containerID="1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"
Jul 06 20:38:08.502837 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.502804    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"} err="failed to get container status \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": rpc error: code = NotFound desc = could not find container \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": container with ID starting with 1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b not found: ID does not exist"
Jul 06 20:38:08.502885 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.502840    2550 scope.go:117] "RemoveContainer" containerID="1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"
Jul 06 20:38:08.503030 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503015    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"} err="failed to get container status \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": rpc error: code = NotFound desc = could not find container \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": container with ID starting with 1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50 not found: ID does not exist"
Jul 06 20:38:08.503085 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503030    2550 scope.go:117] "RemoveContainer" containerID="80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"
Jul 06 20:38:08.503211 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503195    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"} err="failed to get container status \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": rpc error: code = NotFound desc = could not find container \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": container with ID starting with 80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704 not found: ID does not exist"
Jul 06 20:38:08.503254 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503211    2550 scope.go:117] "RemoveContainer" containerID="2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"
Jul 06 20:38:08.503408 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503391    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"} err="failed to get container status \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": rpc error: code = NotFound desc = could not find container \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": container with ID starting with 2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4 not found: ID does not exist"
Jul 06 20:38:08.503449 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503409    2550 scope.go:117] "RemoveContainer" containerID="35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"
Jul 06 20:38:08.503607 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503590    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"} err="failed to get container status \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": rpc error: code = NotFound desc = could not find container \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": container with ID starting with 35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1 not found: ID does not exist"
Jul 06 20:38:08.503644 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503607    2550 scope.go:117] "RemoveContainer" containerID="10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"
Jul 06 20:38:08.503795 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503778    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"} err="failed to get container status \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": rpc error: code = NotFound desc = could not find container \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": container with ID starting with 10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7 not found: ID does not exist"
Jul 06 20:38:08.503864 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503795    2550 scope.go:117] "RemoveContainer" containerID="22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"
Jul 06 20:38:08.504015 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.503998    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"} err="failed to get container status \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": rpc error: code = NotFound desc = could not find container \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": container with ID starting with 22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d not found: ID does not exist"
Jul 06 20:38:08.504015 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.504014    2550 scope.go:117] "RemoveContainer" containerID="1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"
Jul 06 20:38:08.504232 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.504215    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"} err="failed to get container status \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": rpc error: code = NotFound desc = could not find container \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": container with ID starting with 1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b not found: ID does not exist"
Jul 06 20:38:08.504270 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.504234    2550 scope.go:117] "RemoveContainer" containerID="1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"
Jul 06 20:38:08.504442 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.504427    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"} err="failed to get container status \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": rpc error: code = NotFound desc = could not find container \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": container with ID starting with 1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50 not found: ID does not exist"
Jul 06 20:38:08.504478 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.504442    2550 scope.go:117] "RemoveContainer" containerID="80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"
Jul 06 20:38:08.504625 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.504609    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"} err="failed to get container status \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": rpc error: code = NotFound desc = could not find container \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": container with ID starting with 80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704 not found: ID does not exist"
Jul 06 20:38:08.504666 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.504625    2550 scope.go:117] "RemoveContainer" containerID="2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"
Jul 06 20:38:08.504837 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.504800    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"} err="failed to get container status \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": rpc error: code = NotFound desc = could not find container \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": container with ID starting with 2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4 not found: ID does not exist"
Jul 06 20:38:08.504880 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.504839    2550 scope.go:117] "RemoveContainer" containerID="35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"
Jul 06 20:38:08.505061 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.505044    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"} err="failed to get container status \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": rpc error: code = NotFound desc = could not find container \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": container with ID starting with 35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1 not found: ID does not exist"
Jul 06 20:38:08.505100 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.505061    2550 scope.go:117] "RemoveContainer" containerID="10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"
Jul 06 20:38:08.505263 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.505246    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"} err="failed to get container status \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": rpc error: code = NotFound desc = could not find container \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": container with ID starting with 10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7 not found: ID does not exist"
Jul 06 20:38:08.505328 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.505266    2550 scope.go:117] "RemoveContainer" containerID="22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"
Jul 06 20:38:08.505461 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.505446    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"} err="failed to get container status \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": rpc error: code = NotFound desc = could not find container \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": container with ID starting with 22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d not found: ID does not exist"
Jul 06 20:38:08.505500 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.505462    2550 scope.go:117] "RemoveContainer" containerID="1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"
Jul 06 20:38:08.505651 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.505634    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"} err="failed to get container status \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": rpc error: code = NotFound desc = could not find container \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": container with ID starting with 1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b not found: ID does not exist"
Jul 06 20:38:08.505723 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.505652    2550 scope.go:117] "RemoveContainer" containerID="1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"
Jul 06 20:38:08.505899 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.505882    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"} err="failed to get container status \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": rpc error: code = NotFound desc = could not find container \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": container with ID starting with 1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50 not found: ID does not exist"
Jul 06 20:38:08.505952 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.505900    2550 scope.go:117] "RemoveContainer" containerID="80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"
Jul 06 20:38:08.506080 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.506064    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"} err="failed to get container status \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": rpc error: code = NotFound desc = could not find container \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": container with ID starting with 80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704 not found: ID does not exist"
Jul 06 20:38:08.506122 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.506080    2550 scope.go:117] "RemoveContainer" containerID="2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"
Jul 06 20:38:08.506284 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.506260    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"} err="failed to get container status \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": rpc error: code = NotFound desc = could not find container \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": container with ID starting with 2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4 not found: ID does not exist"
Jul 06 20:38:08.506324 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.506284    2550 scope.go:117] "RemoveContainer" containerID="35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"
Jul 06 20:38:08.506465 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.506450    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1"} err="failed to get container status \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": rpc error: code = NotFound desc = could not find container \"35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1\": container with ID starting with 35128ea74f878c3d0da8aa1ebdfc9014e053ff31d7cc2de78f52ce4015a5b1a1 not found: ID does not exist"
Jul 06 20:38:08.506515 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.506465    2550 scope.go:117] "RemoveContainer" containerID="10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"
Jul 06 20:38:08.506631 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.506617    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7"} err="failed to get container status \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": rpc error: code = NotFound desc = could not find container \"10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7\": container with ID starting with 10ab57a27a39ed8d6cfe56fcabca9967fd1cc5b3935e620eaaf383902be486f7 not found: ID does not exist"
Jul 06 20:38:08.506668 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.506632    2550 scope.go:117] "RemoveContainer" containerID="22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"
Jul 06 20:38:08.506819 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.506789    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d"} err="failed to get container status \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": rpc error: code = NotFound desc = could not find container \"22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d\": container with ID starting with 22ed2bc7f6aae3429afeb7a58b6454f4d78db588431d356fa7da382644d18b0d not found: ID does not exist"
Jul 06 20:38:08.506868 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.506843    2550 scope.go:117] "RemoveContainer" containerID="1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"
Jul 06 20:38:08.507041 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.507025    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b"} err="failed to get container status \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": rpc error: code = NotFound desc = could not find container \"1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b\": container with ID starting with 1cfb40bc6570210557774f53a6dbe6750b1bbdcd08568f9487f40efb546f9c0b not found: ID does not exist"
Jul 06 20:38:08.507081 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.507041    2550 scope.go:117] "RemoveContainer" containerID="1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"
Jul 06 20:38:08.507236 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.507219    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50"} err="failed to get container status \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": rpc error: code = NotFound desc = could not find container \"1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50\": container with ID starting with 1f9d9171d787b064129c00226266f6cb884f78ab8ef131eadb9e43d22d376c50 not found: ID does not exist"
Jul 06 20:38:08.507278 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.507235    2550 scope.go:117] "RemoveContainer" containerID="80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"
Jul 06 20:38:08.507563 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.507536    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704"} err="failed to get container status \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": rpc error: code = NotFound desc = could not find container \"80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704\": container with ID starting with 80477247e3f940236f0092989e25c04767278747ae2c486f87d5db2073f5f704 not found: ID does not exist"
Jul 06 20:38:08.507563 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.507553    2550 scope.go:117] "RemoveContainer" containerID="2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"
Jul 06 20:38:08.507838 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.507799    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4"} err="failed to get container status \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": rpc error: code = NotFound desc = could not find container \"2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4\": container with ID starting with 2c5b548df6c7b3821fc620bda038ca16ab83261c9bd57e43ba31775b2a53cfc4 not found: ID does not exist"
Jul 06 20:38:08.518474 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518457    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:38:08.518728 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518717    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy"
Jul 06 20:38:08.518763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518730    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy"
Jul 06 20:38:08.518763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518741    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy"
Jul 06 20:38:08.518763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518747    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy"
Jul 06 20:38:08.518763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518753    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="prom-label-proxy"
Jul 06 20:38:08.518763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518759    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="prom-label-proxy"
Jul 06 20:38:08.518763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518764    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="thanos-sidecar"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518768    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="thanos-sidecar"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518778    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy-metric"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518783    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy-metric"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518790    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354a41da-1c7d-4c43-a17c-bcea3ede2a96" containerName="registry"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518795    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="354a41da-1c7d-4c43-a17c-bcea3ede2a96" containerName="registry"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518803    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="init-config-reloader"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518825    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="init-config-reloader"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518833    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="config-reloader"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518840    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="config-reloader"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518847    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy-thanos"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518854    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy-thanos"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518861    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="config-reloader"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518867    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="config-reloader"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518872    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy-web"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518878    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy-web"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518884    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="alertmanager"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518889    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="alertmanager"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518895    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy-web"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518899    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy-web"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518906    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="init-config-reloader"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518911    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="init-config-reloader"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518917    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="prometheus"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518922    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="prometheus"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518930    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4af1ffc-9cdb-46cc-a339-531fe2c544c5" containerName="console"
Jul 06 20:38:08.518948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518936    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4af1ffc-9cdb-46cc-a339-531fe2c544c5" containerName="console"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518976    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="prom-label-proxy"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518983    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy-web"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518988    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.518996    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4af1ffc-9cdb-46cc-a339-531fe2c544c5" containerName="console"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.519001    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="config-reloader"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.519007    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.519011    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="prometheus"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.519017    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="354a41da-1c7d-4c43-a17c-bcea3ede2a96" containerName="registry"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.519022    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="config-reloader"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.519027    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="alertmanager"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.519031    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="thanos-sidecar"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.519035    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" containerName="kube-rbac-proxy-thanos"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.519040    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy-web"
Jul 06 20:38:08.519586 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.519044    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1735249d-0305-45bd-aefb-3dd636ab9d4e" containerName="kube-rbac-proxy-metric"
Jul 06 20:38:08.524102 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.524082    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.527011 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.526996    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 06 20:38:08.527118 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.527105    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 06 20:38:08.527195 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.527179    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 06 20:38:08.527254 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.527220    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 06 20:38:08.527387 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.527374    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 06 20:38:08.527897 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.527869    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 06 20:38:08.528049 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.528030    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3bj5u64ukmqrl\""
Jul 06 20:38:08.528545 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.528364    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 06 20:38:08.528545 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.528417    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 06 20:38:08.528733 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.528715    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 06 20:38:08.529005 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.528958    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 06 20:38:08.529385 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.529263    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 06 20:38:08.530439 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.529592    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 06 20:38:08.530439 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.530319    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s2tml\""
Jul 06 20:38:08.531783 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.531050    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 06 20:38:08.533412 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.533394    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 06 20:38:08.534520 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.534505    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 06 20:38:08.542189 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.542171    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:38:08.549419 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549402    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549507 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549431    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d65eec04-be37-4369-95ad-74eea6e07f53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549507 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549456    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549507 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549480    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-config\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549534    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549558    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549576    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-578fj\" (UniqueName: \"kubernetes.io/projected/d65eec04-be37-4369-95ad-74eea6e07f53-kube-api-access-578fj\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549601    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549621    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549647    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d65eec04-be37-4369-95ad-74eea6e07f53-config-out\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549662    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549684    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549720    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549757    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549776    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549794    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549832    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549864    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-web-config\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549893    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.549989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.549916    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651181 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651153    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651181 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651183    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-config\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651372 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651201    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651372 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651224    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651372 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651247    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-578fj\" (UniqueName: \"kubernetes.io/projected/d65eec04-be37-4369-95ad-74eea6e07f53-kube-api-access-578fj\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651530 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651372    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651530 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651413    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651530 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651456    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d65eec04-be37-4369-95ad-74eea6e07f53-config-out\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651530 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651481    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651530 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651510    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651764 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651555    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651764 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651586    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651764 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651619    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651764 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651649    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651764 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651673    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651764 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651698    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-web-config\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651764 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651725    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.651764 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651759    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.652164 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651856    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.652164 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651897    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d65eec04-be37-4369-95ad-74eea6e07f53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.652164 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651945    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.652164 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.651945    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.652543 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.652519    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.652782 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.652759    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.654548 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.654450    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.654548 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.654449    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.654701 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.654641    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.655458 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.655281    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d65eec04-be37-4369-95ad-74eea6e07f53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.655547 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.655471    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d65eec04-be37-4369-95ad-74eea6e07f53-config-out\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.656124 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.655611    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.656124 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.655801    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-config\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.656124 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.656084    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.656326 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.656263    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.656384 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.656362    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.656447 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.656392    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.657030 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.657005    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.657113 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.657005    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-web-config\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.657347 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.657324    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d65eec04-be37-4369-95ad-74eea6e07f53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.658141 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.658123    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d65eec04-be37-4369-95ad-74eea6e07f53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.659927 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.659909    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-578fj\" (UniqueName: \"kubernetes.io/projected/d65eec04-be37-4369-95ad-74eea6e07f53-kube-api-access-578fj\") pod \"prometheus-k8s-0\" (UID: \"d65eec04-be37-4369-95ad-74eea6e07f53\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.834756 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.834729    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:08.993858 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:08.993835    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:38:08.995763 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:38:08.995738    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65eec04_be37_4369_95ad_74eea6e07f53.slice/crio-e943a1bf0d5cc12cdb10bf6ec35c4fcdc7ccced2c7f4e329e2842135e522d3dd WatchSource:0}: Error finding container e943a1bf0d5cc12cdb10bf6ec35c4fcdc7ccced2c7f4e329e2842135e522d3dd: Status 404 returned error can't find the container with id e943a1bf0d5cc12cdb10bf6ec35c4fcdc7ccced2c7f4e329e2842135e522d3dd
Jul 06 20:38:09.456531 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:09.456500    2550 generic.go:356] "Generic (PLEG): container finished" podID="d65eec04-be37-4369-95ad-74eea6e07f53" containerID="910ce15489c6d48b7cac85e1005470c3ba55da9e8f9bc948f10e9baeca6e49c8" exitCode=0
Jul 06 20:38:09.456725 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:09.456590    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d65eec04-be37-4369-95ad-74eea6e07f53","Type":"ContainerDied","Data":"910ce15489c6d48b7cac85e1005470c3ba55da9e8f9bc948f10e9baeca6e49c8"}
Jul 06 20:38:09.456725 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:09.456659    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d65eec04-be37-4369-95ad-74eea6e07f53","Type":"ContainerStarted","Data":"e943a1bf0d5cc12cdb10bf6ec35c4fcdc7ccced2c7f4e329e2842135e522d3dd"}
Jul 06 20:38:09.472135 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:09.472112    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003d8026-ce57-4ee3-81b9-e467ff0d590f" path="/var/lib/kubelet/pods/003d8026-ce57-4ee3-81b9-e467ff0d590f/volumes"
Jul 06 20:38:10.463959 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:10.463925    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d65eec04-be37-4369-95ad-74eea6e07f53","Type":"ContainerStarted","Data":"8a0e707b8a79e6c49e5812762ec4eb876872dcbf26839adaef17c9ac461e5278"}
Jul 06 20:38:10.464337 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:10.464014    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d65eec04-be37-4369-95ad-74eea6e07f53","Type":"ContainerStarted","Data":"224095be5fbbdaf277f9a63019f9fdd662da419e6a5a8248e70ca8060c639c6b"}
Jul 06 20:38:10.464337 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:10.464045    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d65eec04-be37-4369-95ad-74eea6e07f53","Type":"ContainerStarted","Data":"54c330e2cf224787ceaf452425cdd33f6d0be9b81454fb50c8fab054f356fd35"}
Jul 06 20:38:10.464337 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:10.464071    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d65eec04-be37-4369-95ad-74eea6e07f53","Type":"ContainerStarted","Data":"17f0bb1ca392f11a6102f39cff42eb1e8b9c59a34e615b08ca316a7c0ebb9a15"}
Jul 06 20:38:10.464337 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:10.464096    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d65eec04-be37-4369-95ad-74eea6e07f53","Type":"ContainerStarted","Data":"e8ccd98bc44a63ba866d4792279577d099226db92c2f7cd455e07a1ff51c8d54"}
Jul 06 20:38:10.464337 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:10.464119    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d65eec04-be37-4369-95ad-74eea6e07f53","Type":"ContainerStarted","Data":"881f81e3a3f961638a65c89c2356e8134dfc07f93831777fa400644448aa7cb9"}
Jul 06 20:38:10.497543 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:10.497499    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.481917705 podStartE2EDuration="2.497484239s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.566534ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:10.495795749 +0000 UTC m=+203.437099607" watchObservedRunningTime="2026-07-06 20:38:10.497484239 +0000 UTC m=+203.438788085"
Jul 06 20:38:13.835037 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:38:13.835006    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:08.835614 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:39:08.835532    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:08.850426 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:39:08.850403    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:09.694361 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:39:09.694334    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:47.369919 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:39:47.369885    2550 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 20:41:56.942820 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.942787    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69f57c9ff6-b62bc"]
Jul 06 20:41:56.945919 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.945899    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:56.950463 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.950428    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 06 20:41:56.950565 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.950551    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 06 20:41:56.951192 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.951173    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 06 20:41:56.951310 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.951208    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 06 20:41:56.951310 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.951220    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 06 20:41:56.951310 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.951231    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n2dlm\""
Jul 06 20:41:56.958560 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.958541    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 06 20:41:56.962091 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.962069    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f57c9ff6-b62bc"]
Jul 06 20:41:56.990276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.990250    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-trusted-ca-bundle\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:56.990409 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.990283    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-service-ca\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:56.990409 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.990304    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4372b006-c5df-4dd2-950f-608658ed33fd-console-serving-cert\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:56.990409 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.990319    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4372b006-c5df-4dd2-950f-608658ed33fd-console-oauth-config\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:56.990409 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.990397    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-console-config\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:56.990564 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.990426    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd299\" (UniqueName: \"kubernetes.io/projected/4372b006-c5df-4dd2-950f-608658ed33fd-kube-api-access-hd299\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:56.990564 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:56.990450    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-oauth-serving-cert\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.091242 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.091203    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-trusted-ca-bundle\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.091242 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.091245    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-service-ca\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.091468 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.091265    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4372b006-c5df-4dd2-950f-608658ed33fd-console-serving-cert\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.091468 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.091280    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4372b006-c5df-4dd2-950f-608658ed33fd-console-oauth-config\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.091468 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.091388    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-console-config\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.091468 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.091421    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd299\" (UniqueName: \"kubernetes.io/projected/4372b006-c5df-4dd2-950f-608658ed33fd-kube-api-access-hd299\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.091468 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.091446    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-oauth-serving-cert\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.092075 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.092049    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-service-ca\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.092075 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.092066    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-console-config\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.092228 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.092095    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-oauth-serving-cert\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.092228 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.092108    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4372b006-c5df-4dd2-950f-608658ed33fd-trusted-ca-bundle\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.093905 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.093884    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4372b006-c5df-4dd2-950f-608658ed33fd-console-serving-cert\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.093989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.093923    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4372b006-c5df-4dd2-950f-608658ed33fd-console-oauth-config\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.099531 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.099508    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd299\" (UniqueName: \"kubernetes.io/projected/4372b006-c5df-4dd2-950f-608658ed33fd-kube-api-access-hd299\") pod \"console-69f57c9ff6-b62bc\" (UID: \"4372b006-c5df-4dd2-950f-608658ed33fd\") " pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.255649 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.255566    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:41:57.395058 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:57.395034    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f57c9ff6-b62bc"]
Jul 06 20:41:57.397294 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:41:57.397267    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4372b006_c5df_4dd2_950f_608658ed33fd.slice/crio-efce1b77ff40bdbeff4d013ed8f5b3ecab09e0d0dda996e5ccb5df73625a272e WatchSource:0}: Error finding container efce1b77ff40bdbeff4d013ed8f5b3ecab09e0d0dda996e5ccb5df73625a272e: Status 404 returned error can't find the container with id efce1b77ff40bdbeff4d013ed8f5b3ecab09e0d0dda996e5ccb5df73625a272e
Jul 06 20:41:58.283500 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:58.283461    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f57c9ff6-b62bc" event={"ID":"4372b006-c5df-4dd2-950f-608658ed33fd","Type":"ContainerStarted","Data":"cfc92a67c53fff2845ee7aef17d8fbe41f9d789d14ae227e7221c991fa4b7868"}
Jul 06 20:41:58.284007 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:58.283520    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f57c9ff6-b62bc" event={"ID":"4372b006-c5df-4dd2-950f-608658ed33fd","Type":"ContainerStarted","Data":"efce1b77ff40bdbeff4d013ed8f5b3ecab09e0d0dda996e5ccb5df73625a272e"}
Jul 06 20:41:58.305616 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:41:58.305573    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69f57c9ff6-b62bc" podStartSLOduration=2.305557913 podStartE2EDuration="2.305557913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:41:58.304517966 +0000 UTC m=+431.245821813" watchObservedRunningTime="2026-07-06 20:41:58.305557913 +0000 UTC m=+431.246861760"
Jul 06 20:42:07.256122 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:07.256074    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:42:07.256670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:07.256187    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:42:07.260865 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:07.260843    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:42:07.319740 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:07.319710    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69f57c9ff6-b62bc"
Jul 06 20:42:12.451910 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.451829    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"]
Jul 06 20:42:12.455025 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.455008    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"
Jul 06 20:42:12.457670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.457643    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9bzbp\""
Jul 06 20:42:12.457786 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.457649    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 06 20:42:12.458630 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.458613    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 06 20:42:12.464115 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.464090    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"]
Jul 06 20:42:12.516525 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.516503    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42b440c-298b-4c82-8134-466780652fe1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fl47s\" (UID: \"a42b440c-298b-4c82-8134-466780652fe1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"
Jul 06 20:42:12.516656 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.516542    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmx72\" (UniqueName: \"kubernetes.io/projected/a42b440c-298b-4c82-8134-466780652fe1-kube-api-access-dmx72\") pod \"cert-manager-cainjector-559cdd8c87-fl47s\" (UID: \"a42b440c-298b-4c82-8134-466780652fe1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"
Jul 06 20:42:12.617396 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.617367    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42b440c-298b-4c82-8134-466780652fe1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fl47s\" (UID: \"a42b440c-298b-4c82-8134-466780652fe1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"
Jul 06 20:42:12.617527 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.617430    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmx72\" (UniqueName: \"kubernetes.io/projected/a42b440c-298b-4c82-8134-466780652fe1-kube-api-access-dmx72\") pod \"cert-manager-cainjector-559cdd8c87-fl47s\" (UID: \"a42b440c-298b-4c82-8134-466780652fe1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"
Jul 06 20:42:12.625608 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.625585    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42b440c-298b-4c82-8134-466780652fe1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fl47s\" (UID: \"a42b440c-298b-4c82-8134-466780652fe1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"
Jul 06 20:42:12.625917 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.625899    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmx72\" (UniqueName: \"kubernetes.io/projected/a42b440c-298b-4c82-8134-466780652fe1-kube-api-access-dmx72\") pod \"cert-manager-cainjector-559cdd8c87-fl47s\" (UID: \"a42b440c-298b-4c82-8134-466780652fe1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"
Jul 06 20:42:12.774414 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.774344    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"
Jul 06 20:42:12.905245 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.905215    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fl47s"]
Jul 06 20:42:12.909446 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:42:12.909416    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda42b440c_298b_4c82_8134_466780652fe1.slice/crio-69e3ad8f7cae18d64602cb20c35f5bf1f5e943e0856dd73445ec61f405724279 WatchSource:0}: Error finding container 69e3ad8f7cae18d64602cb20c35f5bf1f5e943e0856dd73445ec61f405724279: Status 404 returned error can't find the container with id 69e3ad8f7cae18d64602cb20c35f5bf1f5e943e0856dd73445ec61f405724279
Jul 06 20:42:12.911492 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:12.911477    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:42:13.340610 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:13.340578    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s" event={"ID":"a42b440c-298b-4c82-8134-466780652fe1","Type":"ContainerStarted","Data":"69e3ad8f7cae18d64602cb20c35f5bf1f5e943e0856dd73445ec61f405724279"}
Jul 06 20:42:16.357171 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.357137    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s" event={"ID":"a42b440c-298b-4c82-8134-466780652fe1","Type":"ContainerStarted","Data":"6c97663b387ad71125c5849071458df8d2635fc78e25561193f8317913b0bc8a"}
Jul 06 20:42:16.375263 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.375207    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fl47s" podStartSLOduration=1.47861214 podStartE2EDuration="4.375186798s" totalImagesPullingTime="2.896574658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:42:16.373754817 +0000 UTC m=+449.315058665" watchObservedRunningTime="2026-07-06 20:42:16.375186798 +0000 UTC m=+449.316490646"
Jul 06 20:42:16.547183 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.547154    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6bjjr"]
Jul 06 20:42:16.555689 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.555662    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr"
Jul 06 20:42:16.559998 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.559976    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2wxgs\""
Jul 06 20:42:16.565980 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.565959    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6bjjr"]
Jul 06 20:42:16.653716 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.653630    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66715d00-4e82-4660-a669-b6f972cd251c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6bjjr\" (UID: \"66715d00-4e82-4660-a669-b6f972cd251c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr"
Jul 06 20:42:16.653716 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.653684    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bnwv\" (UniqueName: \"kubernetes.io/projected/66715d00-4e82-4660-a669-b6f972cd251c-kube-api-access-7bnwv\") pod \"cert-manager-webhook-6486ffd997-6bjjr\" (UID: \"66715d00-4e82-4660-a669-b6f972cd251c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr"
Jul 06 20:42:16.754117 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.754088    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66715d00-4e82-4660-a669-b6f972cd251c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6bjjr\" (UID: \"66715d00-4e82-4660-a669-b6f972cd251c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr"
Jul 06 20:42:16.754259 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.754133    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnwv\" (UniqueName: \"kubernetes.io/projected/66715d00-4e82-4660-a669-b6f972cd251c-kube-api-access-7bnwv\") pod \"cert-manager-webhook-6486ffd997-6bjjr\" (UID: \"66715d00-4e82-4660-a669-b6f972cd251c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr"
Jul 06 20:42:16.761785 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.761757    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66715d00-4e82-4660-a669-b6f972cd251c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6bjjr\" (UID: \"66715d00-4e82-4660-a669-b6f972cd251c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr"
Jul 06 20:42:16.761918 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.761880    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnwv\" (UniqueName: \"kubernetes.io/projected/66715d00-4e82-4660-a669-b6f972cd251c-kube-api-access-7bnwv\") pod \"cert-manager-webhook-6486ffd997-6bjjr\" (UID: \"66715d00-4e82-4660-a669-b6f972cd251c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr"
Jul 06 20:42:16.864918 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.864890    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr"
Jul 06 20:42:16.993651 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:16.993630    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6bjjr"]
Jul 06 20:42:16.996214 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:42:16.996184    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66715d00_4e82_4660_a669_b6f972cd251c.slice/crio-e1e35f2f0eb0e819ae8438c110833ce3abb6e21992578ee7cffa687a3fc356b5 WatchSource:0}: Error finding container e1e35f2f0eb0e819ae8438c110833ce3abb6e21992578ee7cffa687a3fc356b5: Status 404 returned error can't find the container with id e1e35f2f0eb0e819ae8438c110833ce3abb6e21992578ee7cffa687a3fc356b5
Jul 06 20:42:17.363115 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:17.363080    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr" event={"ID":"66715d00-4e82-4660-a669-b6f972cd251c","Type":"ContainerStarted","Data":"08e4d119ad47acc46abb0fbaa7fe5a0b69a03cd5cf5054a1a7ac1be3282e53f9"}
Jul 06 20:42:17.363115 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:17.363130    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr" event={"ID":"66715d00-4e82-4660-a669-b6f972cd251c","Type":"ContainerStarted","Data":"e1e35f2f0eb0e819ae8438c110833ce3abb6e21992578ee7cffa687a3fc356b5"}
Jul 06 20:42:17.363576 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:17.363183    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr"
Jul 06 20:42:17.388228 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:17.388178    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr" podStartSLOduration=1.388162364 podStartE2EDuration="1.388162364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:42:17.387793847 +0000 UTC m=+450.329097695" watchObservedRunningTime="2026-07-06 20:42:17.388162364 +0000 UTC m=+450.329466210"
Jul 06 20:42:23.369782 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:23.369750    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6bjjr"
Jul 06 20:42:27.210069 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.209982    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"]
Jul 06 20:42:27.213582 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.213560    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"
Jul 06 20:42:27.217229 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.217204    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 06 20:42:27.217356 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.217251    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:42:27.217356 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.217340    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2hwkx\""
Jul 06 20:42:27.224908 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.224888    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"]
Jul 06 20:42:27.334999 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.334955    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/102c9d2e-fb80-4a6c-aedc-5516259870f3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-r25kc\" (UID: \"102c9d2e-fb80-4a6c-aedc-5516259870f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"
Jul 06 20:42:27.334999 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.335000    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m954z\" (UniqueName: \"kubernetes.io/projected/102c9d2e-fb80-4a6c-aedc-5516259870f3-kube-api-access-m954z\") pod \"openshift-lws-operator-fd8ccff4c-r25kc\" (UID: \"102c9d2e-fb80-4a6c-aedc-5516259870f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"
Jul 06 20:42:27.435726 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.435698    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/102c9d2e-fb80-4a6c-aedc-5516259870f3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-r25kc\" (UID: \"102c9d2e-fb80-4a6c-aedc-5516259870f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"
Jul 06 20:42:27.435907 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.435734    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m954z\" (UniqueName: \"kubernetes.io/projected/102c9d2e-fb80-4a6c-aedc-5516259870f3-kube-api-access-m954z\") pod \"openshift-lws-operator-fd8ccff4c-r25kc\" (UID: \"102c9d2e-fb80-4a6c-aedc-5516259870f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"
Jul 06 20:42:27.436159 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.436141    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/102c9d2e-fb80-4a6c-aedc-5516259870f3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-r25kc\" (UID: \"102c9d2e-fb80-4a6c-aedc-5516259870f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"
Jul 06 20:42:27.445095 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.445069    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m954z\" (UniqueName: \"kubernetes.io/projected/102c9d2e-fb80-4a6c-aedc-5516259870f3-kube-api-access-m954z\") pod \"openshift-lws-operator-fd8ccff4c-r25kc\" (UID: \"102c9d2e-fb80-4a6c-aedc-5516259870f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"
Jul 06 20:42:27.522853 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.522781    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"
Jul 06 20:42:27.656405 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:27.656273    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc"]
Jul 06 20:42:27.659063 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:42:27.659031    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod102c9d2e_fb80_4a6c_aedc_5516259870f3.slice/crio-34500d8e730eae129322ad233709d208cc25a481b067b5325598c854b908b527 WatchSource:0}: Error finding container 34500d8e730eae129322ad233709d208cc25a481b067b5325598c854b908b527: Status 404 returned error can't find the container with id 34500d8e730eae129322ad233709d208cc25a481b067b5325598c854b908b527
Jul 06 20:42:28.405605 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:28.405564    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc" event={"ID":"102c9d2e-fb80-4a6c-aedc-5516259870f3","Type":"ContainerStarted","Data":"34500d8e730eae129322ad233709d208cc25a481b067b5325598c854b908b527"}
Jul 06 20:42:30.709997 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:30.709921    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4ktxp"]
Jul 06 20:42:30.713152 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:30.713133    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4ktxp"
Jul 06 20:42:30.715668 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:30.715652    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ghcwm\""
Jul 06 20:42:30.725056 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:30.725034    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4ktxp"]
Jul 06 20:42:30.865290 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:30.865261    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5da8a173-ce4d-48ea-87ca-9e82c2111937-bound-sa-token\") pod \"cert-manager-55b4d954d5-4ktxp\" (UID: \"5da8a173-ce4d-48ea-87ca-9e82c2111937\") " pod="cert-manager/cert-manager-55b4d954d5-4ktxp"
Jul 06 20:42:30.865453 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:30.865302    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxfdb\" (UniqueName: \"kubernetes.io/projected/5da8a173-ce4d-48ea-87ca-9e82c2111937-kube-api-access-hxfdb\") pod \"cert-manager-55b4d954d5-4ktxp\" (UID: \"5da8a173-ce4d-48ea-87ca-9e82c2111937\") " pod="cert-manager/cert-manager-55b4d954d5-4ktxp"
Jul 06 20:42:30.966618 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:30.966546    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5da8a173-ce4d-48ea-87ca-9e82c2111937-bound-sa-token\") pod \"cert-manager-55b4d954d5-4ktxp\" (UID: \"5da8a173-ce4d-48ea-87ca-9e82c2111937\") " pod="cert-manager/cert-manager-55b4d954d5-4ktxp"
Jul 06 20:42:30.966618 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:30.966595    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxfdb\" (UniqueName: \"kubernetes.io/projected/5da8a173-ce4d-48ea-87ca-9e82c2111937-kube-api-access-hxfdb\") pod \"cert-manager-55b4d954d5-4ktxp\" (UID: \"5da8a173-ce4d-48ea-87ca-9e82c2111937\") " pod="cert-manager/cert-manager-55b4d954d5-4ktxp"
Jul 06 20:42:30.975293 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:30.975262    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5da8a173-ce4d-48ea-87ca-9e82c2111937-bound-sa-token\") pod \"cert-manager-55b4d954d5-4ktxp\" (UID: \"5da8a173-ce4d-48ea-87ca-9e82c2111937\") " pod="cert-manager/cert-manager-55b4d954d5-4ktxp"
Jul 06 20:42:30.975403 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:30.975312    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxfdb\" (UniqueName: \"kubernetes.io/projected/5da8a173-ce4d-48ea-87ca-9e82c2111937-kube-api-access-hxfdb\") pod \"cert-manager-55b4d954d5-4ktxp\" (UID: \"5da8a173-ce4d-48ea-87ca-9e82c2111937\") " pod="cert-manager/cert-manager-55b4d954d5-4ktxp"
Jul 06 20:42:31.022279 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:31.022254    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4ktxp"
Jul 06 20:42:31.159142 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:31.159073    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4ktxp"]
Jul 06 20:42:31.161147 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:42:31.161118    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5da8a173_ce4d_48ea_87ca_9e82c2111937.slice/crio-cf17976ef48cc97a78954a6c227048ae640a32d9f5db9ca47310ab5ee18150b7 WatchSource:0}: Error finding container cf17976ef48cc97a78954a6c227048ae640a32d9f5db9ca47310ab5ee18150b7: Status 404 returned error can't find the container with id cf17976ef48cc97a78954a6c227048ae640a32d9f5db9ca47310ab5ee18150b7
Jul 06 20:42:31.419034 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:31.418993    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4ktxp" event={"ID":"5da8a173-ce4d-48ea-87ca-9e82c2111937","Type":"ContainerStarted","Data":"4c1f2783e482f95026d770518a77c169650392171fa60bc6ba5c0890a21a0db1"}
Jul 06 20:42:31.419202 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:31.419048    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4ktxp" event={"ID":"5da8a173-ce4d-48ea-87ca-9e82c2111937","Type":"ContainerStarted","Data":"cf17976ef48cc97a78954a6c227048ae640a32d9f5db9ca47310ab5ee18150b7"}
Jul 06 20:42:31.420617 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:31.420597    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc" event={"ID":"102c9d2e-fb80-4a6c-aedc-5516259870f3","Type":"ContainerStarted","Data":"cb10a1be7ea9b21b82ccdd012235facdba1c6aabd3a7a2a24b5ca71afa9d6104"}
Jul 06 20:42:31.437321 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:31.437277    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4ktxp" podStartSLOduration=1.437264367 podStartE2EDuration="1.437264367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:42:31.435917588 +0000 UTC m=+464.377221448" watchObservedRunningTime="2026-07-06 20:42:31.437264367 +0000 UTC m=+464.378568214"
Jul 06 20:42:31.455257 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:42:31.455210    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r25kc" podStartSLOduration=1.668691429 podStartE2EDuration="4.455196108s" totalImagesPullingTime="2.786504679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:42:31.453104748 +0000 UTC m=+464.394408596" watchObservedRunningTime="2026-07-06 20:42:31.455196108 +0000 UTC m=+464.396499955"
Jul 06 20:43:09.107268 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.107229    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"]
Jul 06 20:43:09.116400 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.116376    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.122038 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.122010    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 06 20:43:09.122170 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.122136    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-24pnd\""
Jul 06 20:43:09.123104 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.123080    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 06 20:43:09.126052 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.123835    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 06 20:43:09.128077 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.128055    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"]
Jul 06 20:43:09.189331 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.189257    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-metrics-cert\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.189525 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.189365    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-cert\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.189525 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.189392    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpj6\" (UniqueName: \"kubernetes.io/projected/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-kube-api-access-9dpj6\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.189525 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.189446    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-manager-config\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.290153 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.290117    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-metrics-cert\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.290348 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.290164    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-cert\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.290348 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.290182    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpj6\" (UniqueName: \"kubernetes.io/projected/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-kube-api-access-9dpj6\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.290348 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.290216    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-manager-config\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.290838 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.290802    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-manager-config\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.292590 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.292572    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-metrics-cert\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.292681 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.292629    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-cert\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.306028 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.306003    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpj6\" (UniqueName: \"kubernetes.io/projected/9a7f455f-ba2e-4ee7-a65c-88607b1e9eff-kube-api-access-9dpj6\") pod \"lws-controller-manager-7845fd6587-tfltn\" (UID: \"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff\") " pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.429096 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.429023    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:09.565897 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:09.565869    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"]
Jul 06 20:43:09.569319 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:43:09.569275    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7f455f_ba2e_4ee7_a65c_88607b1e9eff.slice/crio-c6933df8af67ca81caac49adb1199702c350ef01cef26b2c272dff00a141b86d WatchSource:0}: Error finding container c6933df8af67ca81caac49adb1199702c350ef01cef26b2c272dff00a141b86d: Status 404 returned error can't find the container with id c6933df8af67ca81caac49adb1199702c350ef01cef26b2c272dff00a141b86d
Jul 06 20:43:10.579106 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:10.579064    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn" event={"ID":"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff","Type":"ContainerStarted","Data":"c6933df8af67ca81caac49adb1199702c350ef01cef26b2c272dff00a141b86d"}
Jul 06 20:43:11.584929 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:11.584896    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn" event={"ID":"9a7f455f-ba2e-4ee7-a65c-88607b1e9eff","Type":"ContainerStarted","Data":"3a1c8dbeec1f793dd304d0d13ea0a963ef38f845aae109cad07ac2f32d0779c1"}
Jul 06 20:43:11.585313 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:11.585003    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:43:11.611546 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:11.611498    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn" podStartSLOduration=1.334099266 podStartE2EDuration="2.611482522s" totalImagesPullingTime="1.277383256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:11.608953981 +0000 UTC m=+504.550257851" watchObservedRunningTime="2026-07-06 20:43:11.611482522 +0000 UTC m=+504.552786368"
Jul 06 20:43:22.590464 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:43:22.590432    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7845fd6587-tfltn"
Jul 06 20:44:06.560014 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.559980    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-fsfzc"]
Jul 06 20:44:06.562775 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.562759    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc"
Jul 06 20:44:06.569987 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.569965    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 06 20:44:06.570103 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.569991    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 20:44:06.570738 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.570721    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 20:44:06.572077 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.572059    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fq5bx\""
Jul 06 20:44:06.590987 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.590962    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-fsfzc"]
Jul 06 20:44:06.616288 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.616262    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-fsfzc"]
Jul 06 20:44:06.660578 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.660554    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mnvt\" (UniqueName: \"kubernetes.io/projected/7210fbb3-0495-4e85-bf34-1e768c0b0719-kube-api-access-6mnvt\") pod \"limitador-limitador-69574b596d-fsfzc\" (UID: \"7210fbb3-0495-4e85-bf34-1e768c0b0719\") " pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc"
Jul 06 20:44:06.660701 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.660598    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7210fbb3-0495-4e85-bf34-1e768c0b0719-config-file\") pod \"limitador-limitador-69574b596d-fsfzc\" (UID: \"7210fbb3-0495-4e85-bf34-1e768c0b0719\") " pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc"
Jul 06 20:44:06.761404 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.761372    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mnvt\" (UniqueName: \"kubernetes.io/projected/7210fbb3-0495-4e85-bf34-1e768c0b0719-kube-api-access-6mnvt\") pod \"limitador-limitador-69574b596d-fsfzc\" (UID: \"7210fbb3-0495-4e85-bf34-1e768c0b0719\") " pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc"
Jul 06 20:44:06.761534 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.761416    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7210fbb3-0495-4e85-bf34-1e768c0b0719-config-file\") pod \"limitador-limitador-69574b596d-fsfzc\" (UID: \"7210fbb3-0495-4e85-bf34-1e768c0b0719\") " pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc"
Jul 06 20:44:06.762063 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.762045    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7210fbb3-0495-4e85-bf34-1e768c0b0719-config-file\") pod \"limitador-limitador-69574b596d-fsfzc\" (UID: \"7210fbb3-0495-4e85-bf34-1e768c0b0719\") " pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc"
Jul 06 20:44:06.776049 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.776022    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mnvt\" (UniqueName: \"kubernetes.io/projected/7210fbb3-0495-4e85-bf34-1e768c0b0719-kube-api-access-6mnvt\") pod \"limitador-limitador-69574b596d-fsfzc\" (UID: \"7210fbb3-0495-4e85-bf34-1e768c0b0719\") " pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc"
Jul 06 20:44:06.872772 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:06.872746    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc"
Jul 06 20:44:07.012037 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:07.011926    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-fsfzc"]
Jul 06 20:44:07.014766 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:44:07.014743    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7210fbb3_0495_4e85_bf34_1e768c0b0719.slice/crio-224d76ce1a1b54c18f20ca285da414616a5d61eec627afa8fedd61eaf326ab7e WatchSource:0}: Error finding container 224d76ce1a1b54c18f20ca285da414616a5d61eec627afa8fedd61eaf326ab7e: Status 404 returned error can't find the container with id 224d76ce1a1b54c18f20ca285da414616a5d61eec627afa8fedd61eaf326ab7e
Jul 06 20:44:07.803019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:07.802979    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc" event={"ID":"7210fbb3-0495-4e85-bf34-1e768c0b0719","Type":"ContainerStarted","Data":"224d76ce1a1b54c18f20ca285da414616a5d61eec627afa8fedd61eaf326ab7e"}
Jul 06 20:44:10.819946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:10.819917    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc" event={"ID":"7210fbb3-0495-4e85-bf34-1e768c0b0719","Type":"ContainerStarted","Data":"d159caf714b9a0942494b5d93b361dc053a055a40db79a89e009e035d81b4a6d"}
Jul 06 20:44:10.820326 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:10.820064    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc"
Jul 06 20:44:10.843442 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:10.843396    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc" podStartSLOduration=1.13435255 podStartE2EDuration="4.843381332s" totalImagesPullingTime="3.709028782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:44:10.841744856 +0000 UTC m=+563.783048704" watchObservedRunningTime="2026-07-06 20:44:10.843381332 +0000 UTC m=+563.784685178"
Jul 06 20:44:21.825664 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:44:21.825632    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-69574b596d-fsfzc"
Jul 06 20:47:22.414283 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.414251    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"]
Jul 06 20:47:22.417679 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.417660    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.423286 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.423263    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:47:22.423407 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.423340    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 20:47:22.424359 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.424340    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 20:47:22.424435 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.424340    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 20:47:22.424495 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.424452    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 06 20:47:22.445669 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.445639    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"]
Jul 06 20:47:22.485852 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.485829    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tls-certs\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.485978 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.485876    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-dshm\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.485978 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.485958    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-home\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.486051 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.485994    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4dqq\" (UniqueName: \"kubernetes.io/projected/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kube-api-access-z4dqq\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.486051 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.486030    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-model-cache\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.486051 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.486047    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.486146 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.486070    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.587388 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.587359    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-model-cache\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.587388 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.587389    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.587629 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.587412    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.587629 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.587446    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tls-certs\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.587629 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.587477    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-dshm\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.587629 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.587506    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-home\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.587629 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.587537    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dqq\" (UniqueName: \"kubernetes.io/projected/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kube-api-access-z4dqq\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.588120 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.588095    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.588311 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.588284    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-model-cache\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.588392 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.588225    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.594790 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.594762    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-home\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.595173 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.595129    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tls-certs\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.595974 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.595954    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-dshm\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.597530 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.597508    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dqq\" (UniqueName: \"kubernetes.io/projected/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kube-api-access-z4dqq\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.727965 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.727889    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:22.862552 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.862510    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"]
Jul 06 20:47:22.864768 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:47:22.864742    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2861aa70_c0aa_4fe3_98b6_7be6328b307f.slice/crio-7cf1c7df914f3df11bb01827a8271c36ffd2813828596f11c1521032654f2d2a WatchSource:0}: Error finding container 7cf1c7df914f3df11bb01827a8271c36ffd2813828596f11c1521032654f2d2a: Status 404 returned error can't find the container with id 7cf1c7df914f3df11bb01827a8271c36ffd2813828596f11c1521032654f2d2a
Jul 06 20:47:22.866677 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:22.866661    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:47:23.624422 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:23.624385    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv" event={"ID":"2861aa70-c0aa-4fe3-98b6-7be6328b307f","Type":"ContainerStarted","Data":"7cf1c7df914f3df11bb01827a8271c36ffd2813828596f11c1521032654f2d2a"}
Jul 06 20:47:27.644774 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:27.644738    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv" event={"ID":"2861aa70-c0aa-4fe3-98b6-7be6328b307f","Type":"ContainerStarted","Data":"81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d"}
Jul 06 20:47:31.664492 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:31.664455    2550 generic.go:356] "Generic (PLEG): container finished" podID="2861aa70-c0aa-4fe3-98b6-7be6328b307f" containerID="81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d" exitCode=0
Jul 06 20:47:31.664983 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:31.664509    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv" event={"ID":"2861aa70-c0aa-4fe3-98b6-7be6328b307f","Type":"ContainerDied","Data":"81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d"}
Jul 06 20:47:33.674757 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:33.674725    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv" event={"ID":"2861aa70-c0aa-4fe3-98b6-7be6328b307f","Type":"ContainerStarted","Data":"e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66"}
Jul 06 20:47:33.705789 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:33.705744    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv" podStartSLOduration=2.603024773 podStartE2EDuration="11.705725473s" totalImagesPullingTime="5.355092872s" totalInitContainerRuntime="3.747607828s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:47:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:47:33.704341365 +0000 UTC m=+766.645645215" watchObservedRunningTime="2026-07-06 20:47:33.705725473 +0000 UTC m=+766.647029330"
Jul 06 20:47:33.728324 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:33.728297    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:42.729026 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:42.728995    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:42.741189 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:42.741168    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:43.729207 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:43.729181    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:47:44.867024 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:44.866993    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"]
Jul 06 20:47:44.884502 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:44.884474    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"]
Jul 06 20:47:44.884645 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:44.884579    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:44.887929 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:44.887909    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 06 20:47:44.978875 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:44.978852    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:44.979019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:44.978882    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8drdf\" (UniqueName: \"kubernetes.io/projected/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-kube-api-access-8drdf\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:44.979019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:44.978920    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:44.979019 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:44.978993    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:44.979151 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:44.979018    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:44.979151 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:44.979046    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.079751 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.079719    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.079946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.079764    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.079946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.079786    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.079946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.079859    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.079946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.079925    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.079946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.079945    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8drdf\" (UniqueName: \"kubernetes.io/projected/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-kube-api-access-8drdf\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.080314 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.080275    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.080431 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.080336    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.080431 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.080329    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.082030 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.082010    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.082197 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.082181    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.090169 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.090148    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8drdf\" (UniqueName: \"kubernetes.io/projected/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-kube-api-access-8drdf\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.195292 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.195199    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:45.336086 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.336053    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"]
Jul 06 20:47:45.339023 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:47:45.338995    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377823ff_4a0f_4ea4_8b06_2bdcf25c8231.slice/crio-f39709dbd91abda51472de84cbc08a5960dc67f00ce8d998d8a6e12d52619fa9 WatchSource:0}: Error finding container f39709dbd91abda51472de84cbc08a5960dc67f00ce8d998d8a6e12d52619fa9: Status 404 returned error can't find the container with id f39709dbd91abda51472de84cbc08a5960dc67f00ce8d998d8a6e12d52619fa9
Jul 06 20:47:45.729009 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.728973    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd" event={"ID":"377823ff-4a0f-4ea4-8b06-2bdcf25c8231","Type":"ContainerStarted","Data":"51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183"}
Jul 06 20:47:45.729176 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:45.729036    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd" event={"ID":"377823ff-4a0f-4ea4-8b06-2bdcf25c8231","Type":"ContainerStarted","Data":"f39709dbd91abda51472de84cbc08a5960dc67f00ce8d998d8a6e12d52619fa9"}
Jul 06 20:47:46.195504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:46.195467    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:55.196260 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:55.196223    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:55.208876 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:55.208854    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:47:55.233834 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:55.233780    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd" podStartSLOduration=11.233764655 podStartE2EDuration="11.233764655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:47:45.754556519 +0000 UTC m=+778.695860366" watchObservedRunningTime="2026-07-06 20:47:55.233764655 +0000 UTC m=+788.175068501"
Jul 06 20:47:55.784003 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:47:55.783969    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:48:00.444896 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:00.444865    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"]
Jul 06 20:48:15.464491 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.464391    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd" podUID="377823ff-4a0f-4ea4-8b06-2bdcf25c8231" containerName="main" containerID="cri-o://51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183" gracePeriod=45
Jul 06 20:48:15.715002 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.714946    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:48:15.735960 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.735935    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-home\") pod \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") "
Jul 06 20:48:15.736123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.735998    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-model-cache\") pod \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") "
Jul 06 20:48:15.736123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.736022    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-dshm\") pod \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") "
Jul 06 20:48:15.736123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.736064    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8drdf\" (UniqueName: \"kubernetes.io/projected/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-kube-api-access-8drdf\") pod \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") "
Jul 06 20:48:15.736123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.736113    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tls-certs\") pod \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") "
Jul 06 20:48:15.736383 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.736183    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tmp-dir\") pod \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\" (UID: \"377823ff-4a0f-4ea4-8b06-2bdcf25c8231\") "
Jul 06 20:48:15.736878 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.736654    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "377823ff-4a0f-4ea4-8b06-2bdcf25c8231" (UID: "377823ff-4a0f-4ea4-8b06-2bdcf25c8231"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:15.736878 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.736837    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-home" (OuterVolumeSpecName: "home") pod "377823ff-4a0f-4ea4-8b06-2bdcf25c8231" (UID: "377823ff-4a0f-4ea4-8b06-2bdcf25c8231"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:15.737053 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.737000    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-model-cache" (OuterVolumeSpecName: "model-cache") pod "377823ff-4a0f-4ea4-8b06-2bdcf25c8231" (UID: "377823ff-4a0f-4ea4-8b06-2bdcf25c8231"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:15.739436 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.739398    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "377823ff-4a0f-4ea4-8b06-2bdcf25c8231" (UID: "377823ff-4a0f-4ea4-8b06-2bdcf25c8231"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:48:15.739436 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.739420    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-kube-api-access-8drdf" (OuterVolumeSpecName: "kube-api-access-8drdf") pod "377823ff-4a0f-4ea4-8b06-2bdcf25c8231" (UID: "377823ff-4a0f-4ea4-8b06-2bdcf25c8231"). InnerVolumeSpecName "kube-api-access-8drdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:48:15.739567 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.739463    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-dshm" (OuterVolumeSpecName: "dshm") pod "377823ff-4a0f-4ea4-8b06-2bdcf25c8231" (UID: "377823ff-4a0f-4ea4-8b06-2bdcf25c8231"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:15.837290 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.837253    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:15.837290 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.837296    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:15.837497 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.837305    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:15.837497 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.837314    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:15.837497 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.837323    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8drdf\" (UniqueName: \"kubernetes.io/projected/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-kube-api-access-8drdf\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:15.837497 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.837331    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377823ff-4a0f-4ea4-8b06-2bdcf25c8231-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:15.865236 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.865198    2550 generic.go:356] "Generic (PLEG): container finished" podID="377823ff-4a0f-4ea4-8b06-2bdcf25c8231" containerID="51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183" exitCode=0
Jul 06 20:48:15.865416 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.865247    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"
Jul 06 20:48:15.865416 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.865293    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd" event={"ID":"377823ff-4a0f-4ea4-8b06-2bdcf25c8231","Type":"ContainerDied","Data":"51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183"}
Jul 06 20:48:15.865416 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.865356    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd" event={"ID":"377823ff-4a0f-4ea4-8b06-2bdcf25c8231","Type":"ContainerDied","Data":"f39709dbd91abda51472de84cbc08a5960dc67f00ce8d998d8a6e12d52619fa9"}
Jul 06 20:48:15.865416 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.865385    2550 scope.go:117] "RemoveContainer" containerID="51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183"
Jul 06 20:48:15.874332 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.874316    2550 scope.go:117] "RemoveContainer" containerID="51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183"
Jul 06 20:48:15.874598 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:48:15.874580    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183\": container with ID starting with 51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183 not found: ID does not exist" containerID="51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183"
Jul 06 20:48:15.874644 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.874607    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183"} err="failed to get container status \"51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183\": rpc error: code = NotFound desc = could not find container \"51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183\": container with ID starting with 51992de3d3c79bd3c30e0fc8871a8859ca1141daa9cef83cdaad9f08a1f17183 not found: ID does not exist"
Jul 06 20:48:15.890150 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.890123    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"]
Jul 06 20:48:15.895266 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:15.895245    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd"]
Jul 06 20:48:16.893615 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.893571    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"]
Jul 06 20:48:16.894009 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.893944    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377823ff-4a0f-4ea4-8b06-2bdcf25c8231" containerName="main"
Jul 06 20:48:16.894009 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.893956    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="377823ff-4a0f-4ea4-8b06-2bdcf25c8231" containerName="main"
Jul 06 20:48:16.894009 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.894007    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="377823ff-4a0f-4ea4-8b06-2bdcf25c8231" containerName="main"
Jul 06 20:48:16.897945 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.897927    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:16.901504 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.901487    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 06 20:48:16.916887 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.916854    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"]
Jul 06 20:48:16.946623 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.946591    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:16.946839 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.946704    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:16.946839 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.946747    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:16.946839 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.946777    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpqnn\" (UniqueName: \"kubernetes.io/projected/ae408568-59e7-4b9d-97b0-e216b8e9df8e-kube-api-access-bpqnn\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:16.946839 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.946802    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:16.947042 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:16.946870    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.047515 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.047481    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.047515 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.047522    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpqnn\" (UniqueName: \"kubernetes.io/projected/ae408568-59e7-4b9d-97b0-e216b8e9df8e-kube-api-access-bpqnn\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.047759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.047546    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.047759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.047590    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.047759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.047638    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.047759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.047735    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.048030 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.048009    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.048092 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.048073    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.048147 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.048097    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.049746 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.049724    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.050053 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.050037    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.064393 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.064370    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpqnn\" (UniqueName: \"kubernetes.io/projected/ae408568-59e7-4b9d-97b0-e216b8e9df8e-kube-api-access-bpqnn\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.207235 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.207154    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:17.345782 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.345755    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"]
Jul 06 20:48:17.347796 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:48:17.347770    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae408568_59e7_4b9d_97b0_e216b8e9df8e.slice/crio-f68e1979a7342db62f8457e6d06bb094d683d260e7c4ea95addf3575107f9358 WatchSource:0}: Error finding container f68e1979a7342db62f8457e6d06bb094d683d260e7c4ea95addf3575107f9358: Status 404 returned error can't find the container with id f68e1979a7342db62f8457e6d06bb094d683d260e7c4ea95addf3575107f9358
Jul 06 20:48:17.473957 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.473879    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="377823ff-4a0f-4ea4-8b06-2bdcf25c8231" path="/var/lib/kubelet/pods/377823ff-4a0f-4ea4-8b06-2bdcf25c8231/volumes"
Jul 06 20:48:17.875595 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.875562    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn" event={"ID":"ae408568-59e7-4b9d-97b0-e216b8e9df8e","Type":"ContainerStarted","Data":"b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277"}
Jul 06 20:48:17.875946 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.875616    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn" event={"ID":"ae408568-59e7-4b9d-97b0-e216b8e9df8e","Type":"ContainerStarted","Data":"f68e1979a7342db62f8457e6d06bb094d683d260e7c4ea95addf3575107f9358"}
Jul 06 20:48:17.902892 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:17.902848    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn" podStartSLOduration=1.9028344320000001 podStartE2EDuration="1.902834432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:17.901794214 +0000 UTC m=+810.843098061" watchObservedRunningTime="2026-07-06 20:48:17.902834432 +0000 UTC m=+810.844138279"
Jul 06 20:48:18.207632 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:18.207550    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:20.664386 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:20.664354    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"]
Jul 06 20:48:32.984324 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:32.984294    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"]
Jul 06 20:48:32.988304 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:32.988283    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:32.996935 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:32.996913    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-qc9lf\""
Jul 06 20:48:32.997058 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:32.996972    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 06 20:48:33.017924 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.017902    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"]
Jul 06 20:48:33.086347 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.086318    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.086490 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.086369    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.086490 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.086479    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa118935-4843-4738-ae14-0d7a80c0b0bf-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.086613 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.086530    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgj9j\" (UniqueName: \"kubernetes.io/projected/fa118935-4843-4738-ae14-0d7a80c0b0bf-kube-api-access-wgj9j\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.187417 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.187390    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.187579 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.187439    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.187579 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.187468    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa118935-4843-4738-ae14-0d7a80c0b0bf-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.187579 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.187488    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgj9j\" (UniqueName: \"kubernetes.io/projected/fa118935-4843-4738-ae14-0d7a80c0b0bf-kube-api-access-wgj9j\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.187848 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.187803    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.187917 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.187856    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.189991 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.189964    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa118935-4843-4738-ae14-0d7a80c0b0bf-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.201709 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.201687    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgj9j\" (UniqueName: \"kubernetes.io/projected/fa118935-4843-4738-ae14-0d7a80c0b0bf-kube-api-access-wgj9j\") pod \"auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.298283 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.298192    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:33.441821 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.441781    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"]
Jul 06 20:48:33.443670 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:48:33.443644    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa118935_4843_4738_ae14_0d7a80c0b0bf.slice/crio-b44bb01323d8b4c639a40472e257aa781e53913dd637b16be4adcf4d8c116c05 WatchSource:0}: Error finding container b44bb01323d8b4c639a40472e257aa781e53913dd637b16be4adcf4d8c116c05: Status 404 returned error can't find the container with id b44bb01323d8b4c639a40472e257aa781e53913dd637b16be4adcf4d8c116c05
Jul 06 20:48:33.945706 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.945674    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" event={"ID":"fa118935-4843-4738-ae14-0d7a80c0b0bf","Type":"ContainerStarted","Data":"f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892"}
Jul 06 20:48:33.945875 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:33.945728    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" event={"ID":"fa118935-4843-4738-ae14-0d7a80c0b0bf","Type":"ContainerStarted","Data":"b44bb01323d8b4c639a40472e257aa781e53913dd637b16be4adcf4d8c116c05"}
Jul 06 20:48:34.951089 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:34.951052    2550 generic.go:356] "Generic (PLEG): container finished" podID="fa118935-4843-4738-ae14-0d7a80c0b0bf" containerID="f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892" exitCode=0
Jul 06 20:48:34.951492 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:34.951148    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" event={"ID":"fa118935-4843-4738-ae14-0d7a80c0b0bf","Type":"ContainerDied","Data":"f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892"}
Jul 06 20:48:35.907591 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:35.907530    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn" podUID="ae408568-59e7-4b9d-97b0-e216b8e9df8e" containerName="main" containerID="cri-o://b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277" gracePeriod=45
Jul 06 20:48:36.461032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.461013    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:36.517821 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.517788    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpqnn\" (UniqueName: \"kubernetes.io/projected/ae408568-59e7-4b9d-97b0-e216b8e9df8e-kube-api-access-bpqnn\") pod \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") "
Jul 06 20:48:36.517936 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.517837    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-dshm\") pod \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") "
Jul 06 20:48:36.517936 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.517882    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tls-certs\") pod \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") "
Jul 06 20:48:36.517936 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.517906    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tmp-dir\") pod \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") "
Jul 06 20:48:36.518109 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.517968    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-home\") pod \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") "
Jul 06 20:48:36.518109 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.518039    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-model-cache\") pod \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\" (UID: \"ae408568-59e7-4b9d-97b0-e216b8e9df8e\") "
Jul 06 20:48:36.518206 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.518157    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae408568-59e7-4b9d-97b0-e216b8e9df8e" (UID: "ae408568-59e7-4b9d-97b0-e216b8e9df8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:36.518279 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.518264    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:36.518315 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.518268    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-home" (OuterVolumeSpecName: "home") pod "ae408568-59e7-4b9d-97b0-e216b8e9df8e" (UID: "ae408568-59e7-4b9d-97b0-e216b8e9df8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:36.518353 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.518324    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae408568-59e7-4b9d-97b0-e216b8e9df8e" (UID: "ae408568-59e7-4b9d-97b0-e216b8e9df8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:36.519879 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.519853    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae408568-59e7-4b9d-97b0-e216b8e9df8e-kube-api-access-bpqnn" (OuterVolumeSpecName: "kube-api-access-bpqnn") pod "ae408568-59e7-4b9d-97b0-e216b8e9df8e" (UID: "ae408568-59e7-4b9d-97b0-e216b8e9df8e"). InnerVolumeSpecName "kube-api-access-bpqnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:48:36.520272 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.520256    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-dshm" (OuterVolumeSpecName: "dshm") pod "ae408568-59e7-4b9d-97b0-e216b8e9df8e" (UID: "ae408568-59e7-4b9d-97b0-e216b8e9df8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:36.520335 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.520267    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae408568-59e7-4b9d-97b0-e216b8e9df8e" (UID: "ae408568-59e7-4b9d-97b0-e216b8e9df8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:48:36.619307 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.619280    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae408568-59e7-4b9d-97b0-e216b8e9df8e-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:36.619307 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.619303    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:36.619307 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.619311    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:36.619513 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.619319    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpqnn\" (UniqueName: \"kubernetes.io/projected/ae408568-59e7-4b9d-97b0-e216b8e9df8e-kube-api-access-bpqnn\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:36.619513 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.619328    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae408568-59e7-4b9d-97b0-e216b8e9df8e-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:48:36.965339 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.965302    2550 generic.go:356] "Generic (PLEG): container finished" podID="ae408568-59e7-4b9d-97b0-e216b8e9df8e" containerID="b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277" exitCode=0
Jul 06 20:48:36.965521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.965353    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"
Jul 06 20:48:36.965521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.965390    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn" event={"ID":"ae408568-59e7-4b9d-97b0-e216b8e9df8e","Type":"ContainerDied","Data":"b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277"}
Jul 06 20:48:36.965521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.965448    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn" event={"ID":"ae408568-59e7-4b9d-97b0-e216b8e9df8e","Type":"ContainerDied","Data":"f68e1979a7342db62f8457e6d06bb094d683d260e7c4ea95addf3575107f9358"}
Jul 06 20:48:36.965521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.965483    2550 scope.go:117] "RemoveContainer" containerID="b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277"
Jul 06 20:48:36.967472 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.967450    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" event={"ID":"fa118935-4843-4738-ae14-0d7a80c0b0bf","Type":"ContainerStarted","Data":"f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d"}
Jul 06 20:48:36.967596 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.967513    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:48:36.975670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.975652    2550 scope.go:117] "RemoveContainer" containerID="b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277"
Jul 06 20:48:36.976219 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:48:36.976197    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277\": container with ID starting with b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277 not found: ID does not exist" containerID="b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277"
Jul 06 20:48:36.976361 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.976338    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277"} err="failed to get container status \"b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277\": rpc error: code = NotFound desc = could not find container \"b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277\": container with ID starting with b7061878d49a88fa84c3a98b896e3d0b706740fee97c014b63300bcf43d96277 not found: ID does not exist"
Jul 06 20:48:36.996003 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:36.995954    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" podStartSLOduration=2.822769963 podStartE2EDuration="4.99594019s" totalImagesPullingTime="1.537881347s" totalInitContainerRuntime="635.28888ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:36.995170525 +0000 UTC m=+829.936474372" watchObservedRunningTime="2026-07-06 20:48:36.99594019 +0000 UTC m=+829.937244037"
Jul 06 20:48:37.015861 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:37.015832    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"]
Jul 06 20:48:37.019182 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:37.019163    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn"]
Jul 06 20:48:37.473501 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:37.473467    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae408568-59e7-4b9d-97b0-e216b8e9df8e" path="/var/lib/kubelet/pods/ae408568-59e7-4b9d-97b0-e216b8e9df8e/volumes"
Jul 06 20:48:57.851033 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:48:57.851001    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"]
Jul 06 20:49:07.974141 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:07.974114    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:49:12.870825 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:12.870746    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv" podUID="2861aa70-c0aa-4fe3-98b6-7be6328b307f" containerName="main" containerID="cri-o://e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66" gracePeriod=45
Jul 06 20:49:13.117180 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.117158    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:49:13.122323 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.122256    2550 generic.go:356] "Generic (PLEG): container finished" podID="2861aa70-c0aa-4fe3-98b6-7be6328b307f" containerID="e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66" exitCode=0
Jul 06 20:49:13.122323 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.122306    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"
Jul 06 20:49:13.122323 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.122316    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv" event={"ID":"2861aa70-c0aa-4fe3-98b6-7be6328b307f","Type":"ContainerDied","Data":"e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66"}
Jul 06 20:49:13.122556 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.122398    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv" event={"ID":"2861aa70-c0aa-4fe3-98b6-7be6328b307f","Type":"ContainerDied","Data":"7cf1c7df914f3df11bb01827a8271c36ffd2813828596f11c1521032654f2d2a"}
Jul 06 20:49:13.122556 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.122428    2550 scope.go:117] "RemoveContainer" containerID="e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66"
Jul 06 20:49:13.130490 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.130473    2550 scope.go:117] "RemoveContainer" containerID="81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d"
Jul 06 20:49:13.138035 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.138020    2550 scope.go:117] "RemoveContainer" containerID="e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66"
Jul 06 20:49:13.138284 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:49:13.138267    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66\": container with ID starting with e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66 not found: ID does not exist" containerID="e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66"
Jul 06 20:49:13.138328 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.138294    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66"} err="failed to get container status \"e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66\": rpc error: code = NotFound desc = could not find container \"e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66\": container with ID starting with e5746fd545491b58b4d1a5a200c2bc7fe9954aca18569024c1ca72943b650d66 not found: ID does not exist"
Jul 06 20:49:13.138328 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.138315    2550 scope.go:117] "RemoveContainer" containerID="81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d"
Jul 06 20:49:13.138553 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:49:13.138537    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d\": container with ID starting with 81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d not found: ID does not exist" containerID="81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d"
Jul 06 20:49:13.138593 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.138569    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d"} err="failed to get container status \"81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d\": rpc error: code = NotFound desc = could not find container \"81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d\": container with ID starting with 81e1842c5b8fe9bf4df717397b87910684be966cf2e1865d50af70fc2263ea9d not found: ID does not exist"
Jul 06 20:49:13.214901 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.214878    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4dqq\" (UniqueName: \"kubernetes.io/projected/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kube-api-access-z4dqq\") pod \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") "
Jul 06 20:49:13.215058 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.214913    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kserve-provision-location\") pod \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") "
Jul 06 20:49:13.215058 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.214943    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-model-cache\") pod \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") "
Jul 06 20:49:13.215058 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.214974    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-home\") pod \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") "
Jul 06 20:49:13.215058 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.215021    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-dshm\") pod \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") "
Jul 06 20:49:13.215259 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.215097    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tmp-dir\") pod \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") "
Jul 06 20:49:13.215259 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.215124    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tls-certs\") pod \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\" (UID: \"2861aa70-c0aa-4fe3-98b6-7be6328b307f\") "
Jul 06 20:49:13.215259 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.215230    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2861aa70-c0aa-4fe3-98b6-7be6328b307f" (UID: "2861aa70-c0aa-4fe3-98b6-7be6328b307f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:49:13.215259 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.215248    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-home" (OuterVolumeSpecName: "home") pod "2861aa70-c0aa-4fe3-98b6-7be6328b307f" (UID: "2861aa70-c0aa-4fe3-98b6-7be6328b307f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:49:13.215425 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.215345    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2861aa70-c0aa-4fe3-98b6-7be6328b307f" (UID: "2861aa70-c0aa-4fe3-98b6-7be6328b307f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:49:13.215461 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.215444    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:49:13.215494 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.215464    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:49:13.215494 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.215477    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:49:13.217034 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.217009    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kube-api-access-z4dqq" (OuterVolumeSpecName: "kube-api-access-z4dqq") pod "2861aa70-c0aa-4fe3-98b6-7be6328b307f" (UID: "2861aa70-c0aa-4fe3-98b6-7be6328b307f"). InnerVolumeSpecName "kube-api-access-z4dqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:49:13.217128 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.217099    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-dshm" (OuterVolumeSpecName: "dshm") pod "2861aa70-c0aa-4fe3-98b6-7be6328b307f" (UID: "2861aa70-c0aa-4fe3-98b6-7be6328b307f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:49:13.217276 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.217253    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2861aa70-c0aa-4fe3-98b6-7be6328b307f" (UID: "2861aa70-c0aa-4fe3-98b6-7be6328b307f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:49:13.268396 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.268363    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2861aa70-c0aa-4fe3-98b6-7be6328b307f" (UID: "2861aa70-c0aa-4fe3-98b6-7be6328b307f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:49:13.316453 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.316428    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:49:13.316453 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.316449    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2861aa70-c0aa-4fe3-98b6-7be6328b307f-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:49:13.316572 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.316460    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4dqq\" (UniqueName: \"kubernetes.io/projected/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kube-api-access-z4dqq\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:49:13.316572 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.316473    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2861aa70-c0aa-4fe3-98b6-7be6328b307f-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:49:13.447453 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.447429    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"]
Jul 06 20:49:13.450643 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.450620    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv"]
Jul 06 20:49:13.473364 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:13.473342    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2861aa70-c0aa-4fe3-98b6-7be6328b307f" path="/var/lib/kubelet/pods/2861aa70-c0aa-4fe3-98b6-7be6328b307f/volumes"
Jul 06 20:49:28.664329 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.664297    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"]
Jul 06 20:49:28.664795 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.664766    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2861aa70-c0aa-4fe3-98b6-7be6328b307f" containerName="main"
Jul 06 20:49:28.664795 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.664784    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2861aa70-c0aa-4fe3-98b6-7be6328b307f" containerName="main"
Jul 06 20:49:28.664959 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.664799    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae408568-59e7-4b9d-97b0-e216b8e9df8e" containerName="main"
Jul 06 20:49:28.664959 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.664837    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae408568-59e7-4b9d-97b0-e216b8e9df8e" containerName="main"
Jul 06 20:49:28.664959 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.664850    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2861aa70-c0aa-4fe3-98b6-7be6328b307f" containerName="storage-initializer"
Jul 06 20:49:28.664959 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.664859    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2861aa70-c0aa-4fe3-98b6-7be6328b307f" containerName="storage-initializer"
Jul 06 20:49:28.665138 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.664965    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2861aa70-c0aa-4fe3-98b6-7be6328b307f" containerName="main"
Jul 06 20:49:28.665138 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.664981    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae408568-59e7-4b9d-97b0-e216b8e9df8e" containerName="main"
Jul 06 20:49:28.669289 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.669267    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.671885 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.671866    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 06 20:49:28.680503 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.680482    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"]
Jul 06 20:49:28.743937 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.743904    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.744111 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.743946    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59774c8e-e3f5-4100-a767-e43378a8064e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.744111 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.743983    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.744111 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.744030    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v4q8\" (UniqueName: \"kubernetes.io/projected/59774c8e-e3f5-4100-a767-e43378a8064e-kube-api-access-5v4q8\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.744244 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.744118    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.744244 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.744145    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.844540 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.844498    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59774c8e-e3f5-4100-a767-e43378a8064e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.844716 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.844552    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.844716 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.844581    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v4q8\" (UniqueName: \"kubernetes.io/projected/59774c8e-e3f5-4100-a767-e43378a8064e-kube-api-access-5v4q8\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.844716 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.844638    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.844716 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.844668    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.844956 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.844733    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.845034 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.845013    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.845093 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.845072    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.845209 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.845180    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.846707 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.846684    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.847073 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.847057    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59774c8e-e3f5-4100-a767-e43378a8064e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.854948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.854918    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v4q8\" (UniqueName: \"kubernetes.io/projected/59774c8e-e3f5-4100-a767-e43378a8064e-kube-api-access-5v4q8\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:28.979936 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:28.979858    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:29.113428 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:29.113397    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"]
Jul 06 20:49:29.116327 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:49:29.116302    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59774c8e_e3f5_4100_a767_e43378a8064e.slice/crio-2283720e219a15d6251ed09b9f37c605b8e3da512589b96c0a9b753c364ab77d WatchSource:0}: Error finding container 2283720e219a15d6251ed09b9f37c605b8e3da512589b96c0a9b753c364ab77d: Status 404 returned error can't find the container with id 2283720e219a15d6251ed09b9f37c605b8e3da512589b96c0a9b753c364ab77d
Jul 06 20:49:29.197180 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:29.197146    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d" event={"ID":"59774c8e-e3f5-4100-a767-e43378a8064e","Type":"ContainerStarted","Data":"9ac9514354526432650c55194b6a92dedb5949da86f126f4c983a1e307be92d8"}
Jul 06 20:49:29.197374 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:29.197195    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d" event={"ID":"59774c8e-e3f5-4100-a767-e43378a8064e","Type":"ContainerStarted","Data":"2283720e219a15d6251ed09b9f37c605b8e3da512589b96c0a9b753c364ab77d"}
Jul 06 20:49:29.224428 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:29.224360    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d" podStartSLOduration=1.224336542 podStartE2EDuration="1.224336542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:29.221987623 +0000 UTC m=+882.163291485" watchObservedRunningTime="2026-07-06 20:49:29.224336542 +0000 UTC m=+882.165640391"
Jul 06 20:49:29.980494 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:29.980461    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:38.980218 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:38.980140    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:38.992891 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:38.992865    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:49:39.251912 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:49:39.251846    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:50:02.251759 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:02.251722    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"]
Jul 06 20:50:17.271140 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.271102    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d" podUID="59774c8e-e3f5-4100-a767-e43378a8064e" containerName="main" containerID="cri-o://9ac9514354526432650c55194b6a92dedb5949da86f126f4c983a1e307be92d8" gracePeriod=45
Jul 06 20:50:17.409400 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.409370    2550 generic.go:356] "Generic (PLEG): container finished" podID="59774c8e-e3f5-4100-a767-e43378a8064e" containerID="9ac9514354526432650c55194b6a92dedb5949da86f126f4c983a1e307be92d8" exitCode=0
Jul 06 20:50:17.409560 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.409443    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d" event={"ID":"59774c8e-e3f5-4100-a767-e43378a8064e","Type":"ContainerDied","Data":"9ac9514354526432650c55194b6a92dedb5949da86f126f4c983a1e307be92d8"}
Jul 06 20:50:17.517442 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.517418    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:50:17.644955 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.644931    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-dshm\") pod \"59774c8e-e3f5-4100-a767-e43378a8064e\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") "
Jul 06 20:50:17.645120 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.644968    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-model-cache\") pod \"59774c8e-e3f5-4100-a767-e43378a8064e\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") "
Jul 06 20:50:17.645120 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.645005    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v4q8\" (UniqueName: \"kubernetes.io/projected/59774c8e-e3f5-4100-a767-e43378a8064e-kube-api-access-5v4q8\") pod \"59774c8e-e3f5-4100-a767-e43378a8064e\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") "
Jul 06 20:50:17.645120 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.645024    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-home\") pod \"59774c8e-e3f5-4100-a767-e43378a8064e\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") "
Jul 06 20:50:17.645120 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.645070    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59774c8e-e3f5-4100-a767-e43378a8064e-tls-certs\") pod \"59774c8e-e3f5-4100-a767-e43378a8064e\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") "
Jul 06 20:50:17.645120 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.645115    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-tmp-dir\") pod \"59774c8e-e3f5-4100-a767-e43378a8064e\" (UID: \"59774c8e-e3f5-4100-a767-e43378a8064e\") "
Jul 06 20:50:17.645390 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.645353    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-model-cache" (OuterVolumeSpecName: "model-cache") pod "59774c8e-e3f5-4100-a767-e43378a8064e" (UID: "59774c8e-e3f5-4100-a767-e43378a8064e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:50:17.645453 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.645379    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-home" (OuterVolumeSpecName: "home") pod "59774c8e-e3f5-4100-a767-e43378a8064e" (UID: "59774c8e-e3f5-4100-a767-e43378a8064e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:50:17.645493 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.645469    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59774c8e-e3f5-4100-a767-e43378a8064e" (UID: "59774c8e-e3f5-4100-a767-e43378a8064e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:50:17.647107 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.647087    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-dshm" (OuterVolumeSpecName: "dshm") pod "59774c8e-e3f5-4100-a767-e43378a8064e" (UID: "59774c8e-e3f5-4100-a767-e43378a8064e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:50:17.647228 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.647126    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59774c8e-e3f5-4100-a767-e43378a8064e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59774c8e-e3f5-4100-a767-e43378a8064e" (UID: "59774c8e-e3f5-4100-a767-e43378a8064e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:50:17.647228 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.647209    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59774c8e-e3f5-4100-a767-e43378a8064e-kube-api-access-5v4q8" (OuterVolumeSpecName: "kube-api-access-5v4q8") pod "59774c8e-e3f5-4100-a767-e43378a8064e" (UID: "59774c8e-e3f5-4100-a767-e43378a8064e"). InnerVolumeSpecName "kube-api-access-5v4q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:50:17.746010 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.745988    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:50:17.746010 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.746007    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:50:17.746140 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.746015    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:50:17.746140 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.746024    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v4q8\" (UniqueName: \"kubernetes.io/projected/59774c8e-e3f5-4100-a767-e43378a8064e-kube-api-access-5v4q8\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:50:17.746140 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.746033    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59774c8e-e3f5-4100-a767-e43378a8064e-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:50:17.746140 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:17.746041    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59774c8e-e3f5-4100-a767-e43378a8064e-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:50:18.415299 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:18.415271    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"
Jul 06 20:50:18.416005 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:18.415303    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d" event={"ID":"59774c8e-e3f5-4100-a767-e43378a8064e","Type":"ContainerDied","Data":"2283720e219a15d6251ed09b9f37c605b8e3da512589b96c0a9b753c364ab77d"}
Jul 06 20:50:18.416005 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:18.415365    2550 scope.go:117] "RemoveContainer" containerID="9ac9514354526432650c55194b6a92dedb5949da86f126f4c983a1e307be92d8"
Jul 06 20:50:18.450122 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:18.450101    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"]
Jul 06 20:50:18.456461 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:18.456438    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d"]
Jul 06 20:50:19.473369 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:19.473340    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59774c8e-e3f5-4100-a767-e43378a8064e" path="/var/lib/kubelet/pods/59774c8e-e3f5-4100-a767-e43378a8064e/volumes"
Jul 06 20:50:34.177278 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.177243    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"]
Jul 06 20:50:34.177703 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.177550    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59774c8e-e3f5-4100-a767-e43378a8064e" containerName="main"
Jul 06 20:50:34.177703 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.177561    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="59774c8e-e3f5-4100-a767-e43378a8064e" containerName="main"
Jul 06 20:50:34.177703 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.177633    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="59774c8e-e3f5-4100-a767-e43378a8064e" containerName="main"
Jul 06 20:50:34.180674 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.180658    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.183370 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.183347    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 06 20:50:34.194072 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.194048    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"]
Jul 06 20:50:34.281663 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.281634    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.281872 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.281666    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.281872 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.281691    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001c028b-6c25-4a32-a3e9-241433861563-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.281872 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.281777    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbpnh\" (UniqueName: \"kubernetes.io/projected/001c028b-6c25-4a32-a3e9-241433861563-kube-api-access-nbpnh\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.281872 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.281839    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.282053 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.281880    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.382555 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.382524    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.382700 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.382661    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.382700 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.382678    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.382780 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.382700    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001c028b-6c25-4a32-a3e9-241433861563-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.382780 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.382727    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbpnh\" (UniqueName: \"kubernetes.io/projected/001c028b-6c25-4a32-a3e9-241433861563-kube-api-access-nbpnh\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.382780 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.382746    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.383106 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.383078    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.383191 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.383106    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.383227 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.383208    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.385384 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.385362    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.385590 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.385573    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001c028b-6c25-4a32-a3e9-241433861563-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.392680 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.392646    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbpnh\" (UniqueName: \"kubernetes.io/projected/001c028b-6c25-4a32-a3e9-241433861563-kube-api-access-nbpnh\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.491289 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.491212    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:34.623209 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:34.623185    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"]
Jul 06 20:50:34.625527 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:50:34.625471    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001c028b_6c25_4a32_a3e9_241433861563.slice/crio-012740fbc0071d54add3a8fb39f5a481d9d6a79c6dc08d0b875b422c0defcca3 WatchSource:0}: Error finding container 012740fbc0071d54add3a8fb39f5a481d9d6a79c6dc08d0b875b422c0defcca3: Status 404 returned error can't find the container with id 012740fbc0071d54add3a8fb39f5a481d9d6a79c6dc08d0b875b422c0defcca3
Jul 06 20:50:35.486657 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:35.486623    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj" event={"ID":"001c028b-6c25-4a32-a3e9-241433861563","Type":"ContainerStarted","Data":"b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c"}
Jul 06 20:50:35.487127 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:35.486675    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj" event={"ID":"001c028b-6c25-4a32-a3e9-241433861563","Type":"ContainerStarted","Data":"012740fbc0071d54add3a8fb39f5a481d9d6a79c6dc08d0b875b422c0defcca3"}
Jul 06 20:50:35.491560 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:35.491540    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:35.509078 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:35.509036    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj" podStartSLOduration=1.509023147 podStartE2EDuration="1.509023147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:35.508142746 +0000 UTC m=+948.449446594" watchObservedRunningTime="2026-07-06 20:50:35.509023147 +0000 UTC m=+948.450326994"
Jul 06 20:50:44.491876 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:44.491837    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:44.504210 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:44.504186    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:50:44.540903 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:50:44.540876    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:51:17.929171 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:17.929099    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"]
Jul 06 20:51:17.938979 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:51:17.938940    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 20:51:17.938979 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:51:17.938979 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:51:17.938979 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d" cmd=["/bin/sleep","15"]
Jul 06 20:51:17.939214 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:17.939004    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" message="" err=<
Jul 06 20:51:17.939214 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:51:17.939214 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:51:17.939214 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 20:51:17.939214 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:51:17.939022    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 20:51:17.939214 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:51:17.939214 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:51:17.939214 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" podUID="fa118935-4843-4738-ae14-0d7a80c0b0bf" containerName="main" containerID="cri-o://f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d"
Jul 06 20:51:17.939214 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:17.939078    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" podUID="fa118935-4843-4738-ae14-0d7a80c0b0bf" containerName="main" containerID="cri-o://f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d" gracePeriod=60
Jul 06 20:51:17.973073 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:51:17.973053    2550 logging.go:55] [core] [Channel #104 SubChannel #105]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.32:9003", ServerName: "10.132.0.32:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.32:9003: connect: connection refused"
Jul 06 20:51:18.187305 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.187249    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:51:18.252612 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.252587    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa118935-4843-4738-ae14-0d7a80c0b0bf-tls-certs\") pod \"fa118935-4843-4738-ae14-0d7a80c0b0bf\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") "
Jul 06 20:51:18.252789 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.252630    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-tokenizer-uds\") pod \"fa118935-4843-4738-ae14-0d7a80c0b0bf\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") "
Jul 06 20:51:18.252789 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.252667    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgj9j\" (UniqueName: \"kubernetes.io/projected/fa118935-4843-4738-ae14-0d7a80c0b0bf-kube-api-access-wgj9j\") pod \"fa118935-4843-4738-ae14-0d7a80c0b0bf\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") "
Jul 06 20:51:18.252940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.252791    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-kserve-provision-location\") pod \"fa118935-4843-4738-ae14-0d7a80c0b0bf\" (UID: \"fa118935-4843-4738-ae14-0d7a80c0b0bf\") "
Jul 06 20:51:18.252940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.252922    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "fa118935-4843-4738-ae14-0d7a80c0b0bf" (UID: "fa118935-4843-4738-ae14-0d7a80c0b0bf"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:51:18.253101 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.253083    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:51:18.253408 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.253390    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa118935-4843-4738-ae14-0d7a80c0b0bf" (UID: "fa118935-4843-4738-ae14-0d7a80c0b0bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:51:18.254647 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.254627    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa118935-4843-4738-ae14-0d7a80c0b0bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa118935-4843-4738-ae14-0d7a80c0b0bf" (UID: "fa118935-4843-4738-ae14-0d7a80c0b0bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:51:18.254758 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.254742    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa118935-4843-4738-ae14-0d7a80c0b0bf-kube-api-access-wgj9j" (OuterVolumeSpecName: "kube-api-access-wgj9j") pod "fa118935-4843-4738-ae14-0d7a80c0b0bf" (UID: "fa118935-4843-4738-ae14-0d7a80c0b0bf"). InnerVolumeSpecName "kube-api-access-wgj9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:51:18.353900 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.353863    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa118935-4843-4738-ae14-0d7a80c0b0bf-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:51:18.353900 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.353896    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgj9j\" (UniqueName: \"kubernetes.io/projected/fa118935-4843-4738-ae14-0d7a80c0b0bf-kube-api-access-wgj9j\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:51:18.353900 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.353905    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa118935-4843-4738-ae14-0d7a80c0b0bf-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:51:18.682477 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.682443    2550 generic.go:356] "Generic (PLEG): container finished" podID="fa118935-4843-4738-ae14-0d7a80c0b0bf" containerID="f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d" exitCode=0
Jul 06 20:51:18.682654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.682506    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"
Jul 06 20:51:18.682654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.682523    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" event={"ID":"fa118935-4843-4738-ae14-0d7a80c0b0bf","Type":"ContainerDied","Data":"f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d"}
Jul 06 20:51:18.682654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.682607    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" event={"ID":"fa118935-4843-4738-ae14-0d7a80c0b0bf","Type":"ContainerDied","Data":"b44bb01323d8b4c639a40472e257aa781e53913dd637b16be4adcf4d8c116c05"}
Jul 06 20:51:18.682654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.682633    2550 scope.go:117] "RemoveContainer" containerID="f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d"
Jul 06 20:51:18.691984 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.691966    2550 scope.go:117] "RemoveContainer" containerID="f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892"
Jul 06 20:51:18.699343 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.699328    2550 scope.go:117] "RemoveContainer" containerID="f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d"
Jul 06 20:51:18.699599 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:51:18.699581    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d\": container with ID starting with f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d not found: ID does not exist" containerID="f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d"
Jul 06 20:51:18.699660 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.699606    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d"} err="failed to get container status \"f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d\": rpc error: code = NotFound desc = could not find container \"f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d\": container with ID starting with f416836f22db3a937c1e9d60d34c794e0face120d294ef535ef309259777255d not found: ID does not exist"
Jul 06 20:51:18.699660 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.699622    2550 scope.go:117] "RemoveContainer" containerID="f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892"
Jul 06 20:51:18.699861 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:51:18.699834    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892\": container with ID starting with f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892 not found: ID does not exist" containerID="f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892"
Jul 06 20:51:18.699952 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.699868    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892"} err="failed to get container status \"f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892\": rpc error: code = NotFound desc = could not find container \"f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892\": container with ID starting with f06d86861c0f736eb4b83af7cd7a6a608520ca5b16faee56ce6c8631b2737892 not found: ID does not exist"
Jul 06 20:51:18.705514 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.705494    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"]
Jul 06 20:51:18.711542 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.711524    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn"]
Jul 06 20:51:18.973111 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:18.973015    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn" podUID="fa118935-4843-4738-ae14-0d7a80c0b0bf" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.32:9003\" within 1s: context deadline exceeded"
Jul 06 20:51:19.474453 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:19.474424    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa118935-4843-4738-ae14-0d7a80c0b0bf" path="/var/lib/kubelet/pods/fa118935-4843-4738-ae14-0d7a80c0b0bf/volumes"
Jul 06 20:51:50.543114 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.543083    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"]
Jul 06 20:51:50.543623 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.543564    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa118935-4843-4738-ae14-0d7a80c0b0bf" containerName="main"
Jul 06 20:51:50.543623 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.543582    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa118935-4843-4738-ae14-0d7a80c0b0bf" containerName="main"
Jul 06 20:51:50.543623 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.543596    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa118935-4843-4738-ae14-0d7a80c0b0bf" containerName="storage-initializer"
Jul 06 20:51:50.543623 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.543605    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa118935-4843-4738-ae14-0d7a80c0b0bf" containerName="storage-initializer"
Jul 06 20:51:50.543857 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.543686    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa118935-4843-4738-ae14-0d7a80c0b0bf" containerName="main"
Jul 06 20:51:50.547657 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.547637    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:50.551731 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.551712    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 06 20:51:50.552845 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.552803    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-dr645\""
Jul 06 20:51:50.560990 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.560968    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"]
Jul 06 20:51:50.619930 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.619905    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:50.620061 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.619938    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrx6p\" (UniqueName: \"kubernetes.io/projected/1865b8f3-c758-4ddc-ab3c-cf28d089617f-kube-api-access-mrx6p\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:50.620061 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.619963    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:50.721043 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.721009    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:50.721043 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.721045    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrx6p\" (UniqueName: \"kubernetes.io/projected/1865b8f3-c758-4ddc-ab3c-cf28d089617f-kube-api-access-mrx6p\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:50.721244 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.721075    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:50.721452 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.721432    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:50.723438 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.723421    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:50.741624 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.741599    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrx6p\" (UniqueName: \"kubernetes.io/projected/1865b8f3-c758-4ddc-ab3c-cf28d089617f-kube-api-access-mrx6p\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:50.857374 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:50.857351    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:51.009970 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:51.009948    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"]
Jul 06 20:51:51.011013 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:51:51.010990    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1865b8f3_c758_4ddc_ab3c_cf28d089617f.slice/crio-3fc0de857a9e5046a43860133db72dcf4440a5c0279144fb2d4fb8eca8cbd9d8 WatchSource:0}: Error finding container 3fc0de857a9e5046a43860133db72dcf4440a5c0279144fb2d4fb8eca8cbd9d8: Status 404 returned error can't find the container with id 3fc0de857a9e5046a43860133db72dcf4440a5c0279144fb2d4fb8eca8cbd9d8
Jul 06 20:51:51.826098 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:51.826058    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b" event={"ID":"1865b8f3-c758-4ddc-ab3c-cf28d089617f","Type":"ContainerStarted","Data":"8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9"}
Jul 06 20:51:51.826562 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:51.826110    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b" event={"ID":"1865b8f3-c758-4ddc-ab3c-cf28d089617f","Type":"ContainerStarted","Data":"3fc0de857a9e5046a43860133db72dcf4440a5c0279144fb2d4fb8eca8cbd9d8"}
Jul 06 20:51:51.826562 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:51.826149    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:51:51.853838 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:51:51.853780    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b" podStartSLOduration=1.853766393 podStartE2EDuration="1.853766393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:51:51.851039146 +0000 UTC m=+1024.792342994" watchObservedRunningTime="2026-07-06 20:51:51.853766393 +0000 UTC m=+1024.795070239"
Jul 06 20:52:01.107481 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.107446    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"]
Jul 06 20:52:01.111480 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.111459    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.117009 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.116989    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:52:01.117539 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.117524    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 06 20:52:01.118962 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.118604    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-hh644\""
Jul 06 20:52:01.130234 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.130213    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"]
Jul 06 20:52:01.213111 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.213079    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4n6\" (UniqueName: \"kubernetes.io/projected/453f51b1-e436-4038-bf28-4b129b2ae05e-kube-api-access-kn4n6\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.213265 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.213134    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/453f51b1-e436-4038-bf28-4b129b2ae05e-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.213265 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.213208    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.213265 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.213260    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.314091 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.314066    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/453f51b1-e436-4038-bf28-4b129b2ae05e-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.314220 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.314113    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.314287 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.314236    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.314346 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.314307    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4n6\" (UniqueName: \"kubernetes.io/projected/453f51b1-e436-4038-bf28-4b129b2ae05e-kube-api-access-kn4n6\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.314432 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.314410    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.314573 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.314555    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.316452 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.316432    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/453f51b1-e436-4038-bf28-4b129b2ae05e-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.323251 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.323233    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4n6\" (UniqueName: \"kubernetes.io/projected/453f51b1-e436-4038-bf28-4b129b2ae05e-kube-api-access-kn4n6\") pod \"auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.421343 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.421267    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:01.565383 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.565359    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"]
Jul 06 20:52:01.567253 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:52:01.567224    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453f51b1_e436_4038_bf28_4b129b2ae05e.slice/crio-664b78822d7959fb38d84f418a4e48cf9b15e5d7e9f8d9be385f50fe2c472cbc WatchSource:0}: Error finding container 664b78822d7959fb38d84f418a4e48cf9b15e5d7e9f8d9be385f50fe2c472cbc: Status 404 returned error can't find the container with id 664b78822d7959fb38d84f418a4e48cf9b15e5d7e9f8d9be385f50fe2c472cbc
Jul 06 20:52:01.869930 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.869891    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp" event={"ID":"453f51b1-e436-4038-bf28-4b129b2ae05e","Type":"ContainerStarted","Data":"676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc"}
Jul 06 20:52:01.870120 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:01.869950    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp" event={"ID":"453f51b1-e436-4038-bf28-4b129b2ae05e","Type":"ContainerStarted","Data":"664b78822d7959fb38d84f418a4e48cf9b15e5d7e9f8d9be385f50fe2c472cbc"}
Jul 06 20:52:02.875552 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:02.875517    2550 generic.go:356] "Generic (PLEG): container finished" podID="453f51b1-e436-4038-bf28-4b129b2ae05e" containerID="676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc" exitCode=0
Jul 06 20:52:02.875972 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:02.875590    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp" event={"ID":"453f51b1-e436-4038-bf28-4b129b2ae05e","Type":"ContainerDied","Data":"676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc"}
Jul 06 20:52:03.881933 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:03.881894    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp" event={"ID":"453f51b1-e436-4038-bf28-4b129b2ae05e","Type":"ContainerStarted","Data":"e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087"}
Jul 06 20:52:03.882292 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:03.882029    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:22.832971 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:22.832940    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:52:22.863502 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:22.863450    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp" podStartSLOduration=21.20509764 podStartE2EDuration="21.863435045s" totalImagesPullingTime="0s" totalInitContainerRuntime="658.337405ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:52:03.914997927 +0000 UTC m=+1036.856301773" watchObservedRunningTime="2026-07-06 20:52:22.863435045 +0000 UTC m=+1055.804738891"
Jul 06 20:52:34.888134 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:34.888107    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:52:55.721193 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:55.721161    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"]
Jul 06 20:52:55.729750 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:52:55.729714    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 20:52:55.729750 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:52:55.729750 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:52:55.729750 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9" cmd=["/bin/sleep","15"]
Jul 06 20:52:55.730005 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:55.729765    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b" message="" err=<
Jul 06 20:52:55.730005 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:52:55.730005 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:52:55.730005 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 20:52:55.730005 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:52:55.729777    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 20:52:55.730005 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:52:55.730005 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:52:55.730005 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b" podUID="1865b8f3-c758-4ddc-ab3c-cf28d089617f" containerName="main" containerID="cri-o://8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9"
Jul 06 20:52:55.730005 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:55.729835    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b" podUID="1865b8f3-c758-4ddc-ab3c-cf28d089617f" containerName="main" containerID="cri-o://8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9" gracePeriod=60
Jul 06 20:52:55.972194 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:55.972136    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:52:56.075682 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.075655    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tokenizer-uds\") pod \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") "
Jul 06 20:52:56.075883 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.075773    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrx6p\" (UniqueName: \"kubernetes.io/projected/1865b8f3-c758-4ddc-ab3c-cf28d089617f-kube-api-access-mrx6p\") pod \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") "
Jul 06 20:52:56.075883 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.075824    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tls-certs\") pod \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\" (UID: \"1865b8f3-c758-4ddc-ab3c-cf28d089617f\") "
Jul 06 20:52:56.076005 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.075928    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1865b8f3-c758-4ddc-ab3c-cf28d089617f" (UID: "1865b8f3-c758-4ddc-ab3c-cf28d089617f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:52:56.076092 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.076076    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:52:56.077775 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.077748    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1865b8f3-c758-4ddc-ab3c-cf28d089617f" (UID: "1865b8f3-c758-4ddc-ab3c-cf28d089617f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:56.077893 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.077754    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1865b8f3-c758-4ddc-ab3c-cf28d089617f-kube-api-access-mrx6p" (OuterVolumeSpecName: "kube-api-access-mrx6p") pod "1865b8f3-c758-4ddc-ab3c-cf28d089617f" (UID: "1865b8f3-c758-4ddc-ab3c-cf28d089617f"). InnerVolumeSpecName "kube-api-access-mrx6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:52:56.114711 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.114682    2550 generic.go:356] "Generic (PLEG): container finished" podID="1865b8f3-c758-4ddc-ab3c-cf28d089617f" containerID="8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9" exitCode=0
Jul 06 20:52:56.114845 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.114723    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"
Jul 06 20:52:56.114845 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.114737    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b" event={"ID":"1865b8f3-c758-4ddc-ab3c-cf28d089617f","Type":"ContainerDied","Data":"8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9"}
Jul 06 20:52:56.114845 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.114784    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b" event={"ID":"1865b8f3-c758-4ddc-ab3c-cf28d089617f","Type":"ContainerDied","Data":"3fc0de857a9e5046a43860133db72dcf4440a5c0279144fb2d4fb8eca8cbd9d8"}
Jul 06 20:52:56.114845 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.114818    2550 scope.go:117] "RemoveContainer" containerID="8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9"
Jul 06 20:52:56.123801 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.123785    2550 scope.go:117] "RemoveContainer" containerID="8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9"
Jul 06 20:52:56.124113 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:52:56.124094    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9\": container with ID starting with 8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9 not found: ID does not exist" containerID="8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9"
Jul 06 20:52:56.124161 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.124120    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9"} err="failed to get container status \"8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9\": rpc error: code = NotFound desc = could not find container \"8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9\": container with ID starting with 8f764c801df9734399db33e333eeafdd6e431387f8142c00792c49d4b4a0d8e9 not found: ID does not exist"
Jul 06 20:52:56.137387 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.137361    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"]
Jul 06 20:52:56.142898 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.142877    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b"]
Jul 06 20:52:56.176708 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.176678    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrx6p\" (UniqueName: \"kubernetes.io/projected/1865b8f3-c758-4ddc-ab3c-cf28d089617f-kube-api-access-mrx6p\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:52:56.176708 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:56.176707    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1865b8f3-c758-4ddc-ab3c-cf28d089617f-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:52:57.475285 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:52:57.475250    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1865b8f3-c758-4ddc-ab3c-cf28d089617f" path="/var/lib/kubelet/pods/1865b8f3-c758-4ddc-ab3c-cf28d089617f/volumes"
Jul 06 20:53:12.810338 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:12.810303    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"]
Jul 06 20:53:27.830236 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:27.830196    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj" podUID="001c028b-6c25-4a32-a3e9-241433861563" containerName="main" containerID="cri-o://b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c" gracePeriod=45
Jul 06 20:53:28.069294 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.069268    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:53:28.255454 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.255378    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-dshm\") pod \"001c028b-6c25-4a32-a3e9-241433861563\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") "
Jul 06 20:53:28.255454 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.255448    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001c028b-6c25-4a32-a3e9-241433861563-tls-certs\") pod \"001c028b-6c25-4a32-a3e9-241433861563\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") "
Jul 06 20:53:28.255667 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.255498    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-home\") pod \"001c028b-6c25-4a32-a3e9-241433861563\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") "
Jul 06 20:53:28.255667 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.255525    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbpnh\" (UniqueName: \"kubernetes.io/projected/001c028b-6c25-4a32-a3e9-241433861563-kube-api-access-nbpnh\") pod \"001c028b-6c25-4a32-a3e9-241433861563\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") "
Jul 06 20:53:28.255667 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.255549    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-model-cache\") pod \"001c028b-6c25-4a32-a3e9-241433861563\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") "
Jul 06 20:53:28.255667 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.255575    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-tmp-dir\") pod \"001c028b-6c25-4a32-a3e9-241433861563\" (UID: \"001c028b-6c25-4a32-a3e9-241433861563\") "
Jul 06 20:53:28.255883 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.255775    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-home" (OuterVolumeSpecName: "home") pod "001c028b-6c25-4a32-a3e9-241433861563" (UID: "001c028b-6c25-4a32-a3e9-241433861563"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:53:28.255883 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.255803    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-model-cache" (OuterVolumeSpecName: "model-cache") pod "001c028b-6c25-4a32-a3e9-241433861563" (UID: "001c028b-6c25-4a32-a3e9-241433861563"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:53:28.256014 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.255986    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "001c028b-6c25-4a32-a3e9-241433861563" (UID: "001c028b-6c25-4a32-a3e9-241433861563"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:53:28.257436 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.257411    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-dshm" (OuterVolumeSpecName: "dshm") pod "001c028b-6c25-4a32-a3e9-241433861563" (UID: "001c028b-6c25-4a32-a3e9-241433861563"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:53:28.257634 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.257611    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001c028b-6c25-4a32-a3e9-241433861563-kube-api-access-nbpnh" (OuterVolumeSpecName: "kube-api-access-nbpnh") pod "001c028b-6c25-4a32-a3e9-241433861563" (UID: "001c028b-6c25-4a32-a3e9-241433861563"). InnerVolumeSpecName "kube-api-access-nbpnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:53:28.257721 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.257641    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001c028b-6c25-4a32-a3e9-241433861563-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "001c028b-6c25-4a32-a3e9-241433861563" (UID: "001c028b-6c25-4a32-a3e9-241433861563"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:53:28.262564 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.262541    2550 generic.go:356] "Generic (PLEG): container finished" podID="001c028b-6c25-4a32-a3e9-241433861563" containerID="b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c" exitCode=0
Jul 06 20:53:28.262686 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.262577    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj" event={"ID":"001c028b-6c25-4a32-a3e9-241433861563","Type":"ContainerDied","Data":"b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c"}
Jul 06 20:53:28.262686 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.262585    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"
Jul 06 20:53:28.262686 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.262630    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj" event={"ID":"001c028b-6c25-4a32-a3e9-241433861563","Type":"ContainerDied","Data":"012740fbc0071d54add3a8fb39f5a481d9d6a79c6dc08d0b875b422c0defcca3"}
Jul 06 20:53:28.262686 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.262661    2550 scope.go:117] "RemoveContainer" containerID="b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c"
Jul 06 20:53:28.273940 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.273924    2550 scope.go:117] "RemoveContainer" containerID="b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c"
Jul 06 20:53:28.274185 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:53:28.274166    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c\": container with ID starting with b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c not found: ID does not exist" containerID="b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c"
Jul 06 20:53:28.274248 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.274193    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c"} err="failed to get container status \"b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c\": rpc error: code = NotFound desc = could not find container \"b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c\": container with ID starting with b168432d3e8688ff53f84a29ff40db61d2e3d70c627df9c6652df0cc25b8c70c not found: ID does not exist"
Jul 06 20:53:28.285976 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.285956    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"]
Jul 06 20:53:28.290777 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.290758    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj"]
Jul 06 20:53:28.356267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.356236    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001c028b-6c25-4a32-a3e9-241433861563-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:53:28.356267 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.356269    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:53:28.356417 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.356278    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbpnh\" (UniqueName: \"kubernetes.io/projected/001c028b-6c25-4a32-a3e9-241433861563-kube-api-access-nbpnh\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:53:28.356417 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.356289    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:53:28.356417 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.356297    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:53:28.356417 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:28.356304    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/001c028b-6c25-4a32-a3e9-241433861563-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:53:29.473187 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:29.473159    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="001c028b-6c25-4a32-a3e9-241433861563" path="/var/lib/kubelet/pods/001c028b-6c25-4a32-a3e9-241433861563/volumes"
Jul 06 20:53:45.367868 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.367840    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"]
Jul 06 20:53:45.368308 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.368194    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1865b8f3-c758-4ddc-ab3c-cf28d089617f" containerName="main"
Jul 06 20:53:45.368308 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.368205    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1865b8f3-c758-4ddc-ab3c-cf28d089617f" containerName="main"
Jul 06 20:53:45.368308 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.368214    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001c028b-6c25-4a32-a3e9-241433861563" containerName="main"
Jul 06 20:53:45.368308 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.368220    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="001c028b-6c25-4a32-a3e9-241433861563" containerName="main"
Jul 06 20:53:45.368308 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.368271    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="001c028b-6c25-4a32-a3e9-241433861563" containerName="main"
Jul 06 20:53:45.368308 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.368279    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1865b8f3-c758-4ddc-ab3c-cf28d089617f" containerName="main"
Jul 06 20:53:45.371123 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.371108    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.373869 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.373834    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 06 20:53:45.381461 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.381441    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"]
Jul 06 20:53:45.492989 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.492962    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.493144 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.492994    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x7zp\" (UniqueName: \"kubernetes.io/projected/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-kube-api-access-6x7zp\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.493144 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.493016    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.493144 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.493060    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.493144 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.493125    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.493144 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.493142    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.593796 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.593767    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.593978 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.593802    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7zp\" (UniqueName: \"kubernetes.io/projected/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-kube-api-access-6x7zp\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.593978 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.593844    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.593978 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.593864    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.593978 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.593916    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.593978 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.593934    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.594229 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.594170    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.594358 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.594332    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.594358 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.594350    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.596204 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.596176    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.596458 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.596441    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.603799 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.603775    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7zp\" (UniqueName: \"kubernetes.io/projected/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-kube-api-access-6x7zp\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.681544 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.681475    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:45.814337 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:45.814305    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"]
Jul 06 20:53:45.817195 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:53:45.817167    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a7b61f_8ad8_4240_aaa7_08de4ac64a15.slice/crio-fd7f5464412ad09527d1d2cb6a24531b3f50f8526a87204b940be224a17953b9 WatchSource:0}: Error finding container fd7f5464412ad09527d1d2cb6a24531b3f50f8526a87204b940be224a17953b9: Status 404 returned error can't find the container with id fd7f5464412ad09527d1d2cb6a24531b3f50f8526a87204b940be224a17953b9
Jul 06 20:53:46.341406 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:46.341373    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9" event={"ID":"62a7b61f-8ad8-4240-aaa7-08de4ac64a15","Type":"ContainerStarted","Data":"02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9"}
Jul 06 20:53:46.341581 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:46.341425    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9" event={"ID":"62a7b61f-8ad8-4240-aaa7-08de4ac64a15","Type":"ContainerStarted","Data":"fd7f5464412ad09527d1d2cb6a24531b3f50f8526a87204b940be224a17953b9"}
Jul 06 20:53:46.365027 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:46.364981    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9" podStartSLOduration=1.364966989 podStartE2EDuration="1.364966989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:53:46.363350155 +0000 UTC m=+1139.304654002" watchObservedRunningTime="2026-07-06 20:53:46.364966989 +0000 UTC m=+1139.306270883"
Jul 06 20:53:46.682080 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:46.682001    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:55.682583 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:55.682545    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:55.694963 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:55.694938    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:53:56.399576 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:53:56.399547    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:54:02.838705 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:02.838674    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"]
Jul 06 20:54:02.848883 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:54:02.848849    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 20:54:02.848883 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:54:02.848883 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:54:02.848883 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087" cmd=["/bin/sleep","15"]
Jul 06 20:54:02.849198 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:02.848896    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp" message="" err=<
Jul 06 20:54:02.849198 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:54:02.849198 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:54:02.849198 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 20:54:02.849198 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:54:02.848910    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 20:54:02.849198 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:54:02.849198 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:54:02.849198 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp" podUID="453f51b1-e436-4038-bf28-4b129b2ae05e" containerName="main" containerID="cri-o://e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087"
Jul 06 20:54:02.849198 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:02.848964    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp" podUID="453f51b1-e436-4038-bf28-4b129b2ae05e" containerName="main" containerID="cri-o://e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087" gracePeriod=60
Jul 06 20:54:03.094899 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.094842    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:54:03.241730 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.241698    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn4n6\" (UniqueName: \"kubernetes.io/projected/453f51b1-e436-4038-bf28-4b129b2ae05e-kube-api-access-kn4n6\") pod \"453f51b1-e436-4038-bf28-4b129b2ae05e\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") "
Jul 06 20:54:03.241915 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.241744    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-tokenizer-uds\") pod \"453f51b1-e436-4038-bf28-4b129b2ae05e\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") "
Jul 06 20:54:03.241915 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.241776    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-kserve-provision-location\") pod \"453f51b1-e436-4038-bf28-4b129b2ae05e\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") "
Jul 06 20:54:03.242053 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.241920    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/453f51b1-e436-4038-bf28-4b129b2ae05e-tls-certs\") pod \"453f51b1-e436-4038-bf28-4b129b2ae05e\" (UID: \"453f51b1-e436-4038-bf28-4b129b2ae05e\") "
Jul 06 20:54:03.242114 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.242045    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "453f51b1-e436-4038-bf28-4b129b2ae05e" (UID: "453f51b1-e436-4038-bf28-4b129b2ae05e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:54:03.242242 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.242223    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:54:03.242511 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.242493    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "453f51b1-e436-4038-bf28-4b129b2ae05e" (UID: "453f51b1-e436-4038-bf28-4b129b2ae05e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:54:03.243925 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.243901    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453f51b1-e436-4038-bf28-4b129b2ae05e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "453f51b1-e436-4038-bf28-4b129b2ae05e" (UID: "453f51b1-e436-4038-bf28-4b129b2ae05e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:54:03.244029 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.243922    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453f51b1-e436-4038-bf28-4b129b2ae05e-kube-api-access-kn4n6" (OuterVolumeSpecName: "kube-api-access-kn4n6") pod "453f51b1-e436-4038-bf28-4b129b2ae05e" (UID: "453f51b1-e436-4038-bf28-4b129b2ae05e"). InnerVolumeSpecName "kube-api-access-kn4n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:54:03.342654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.342627    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453f51b1-e436-4038-bf28-4b129b2ae05e-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:54:03.342654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.342649    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/453f51b1-e436-4038-bf28-4b129b2ae05e-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:54:03.342654 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.342660    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn4n6\" (UniqueName: \"kubernetes.io/projected/453f51b1-e436-4038-bf28-4b129b2ae05e-kube-api-access-kn4n6\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:54:03.423843 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.423761    2550 generic.go:356] "Generic (PLEG): container finished" podID="453f51b1-e436-4038-bf28-4b129b2ae05e" containerID="e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087" exitCode=0
Jul 06 20:54:03.423843 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.423794    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp" event={"ID":"453f51b1-e436-4038-bf28-4b129b2ae05e","Type":"ContainerDied","Data":"e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087"}
Jul 06 20:54:03.423843 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.423821    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"
Jul 06 20:54:03.424038 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.423847    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp" event={"ID":"453f51b1-e436-4038-bf28-4b129b2ae05e","Type":"ContainerDied","Data":"664b78822d7959fb38d84f418a4e48cf9b15e5d7e9f8d9be385f50fe2c472cbc"}
Jul 06 20:54:03.424038 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.423867    2550 scope.go:117] "RemoveContainer" containerID="e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087"
Jul 06 20:54:03.432575 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.432559    2550 scope.go:117] "RemoveContainer" containerID="676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc"
Jul 06 20:54:03.439667 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.439652    2550 scope.go:117] "RemoveContainer" containerID="e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087"
Jul 06 20:54:03.439919 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:54:03.439899    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087\": container with ID starting with e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087 not found: ID does not exist" containerID="e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087"
Jul 06 20:54:03.439983 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.439925    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087"} err="failed to get container status \"e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087\": rpc error: code = NotFound desc = could not find container \"e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087\": container with ID starting with e72e6a77788879e0e4e34045bb558c2c8753cb734f70909542dfcc9d9eeb4087 not found: ID does not exist"
Jul 06 20:54:03.439983 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.439942    2550 scope.go:117] "RemoveContainer" containerID="676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc"
Jul 06 20:54:03.440171 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:54:03.440153    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc\": container with ID starting with 676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc not found: ID does not exist" containerID="676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc"
Jul 06 20:54:03.440210 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.440176    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc"} err="failed to get container status \"676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc\": rpc error: code = NotFound desc = could not find container \"676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc\": container with ID starting with 676484d7d3fab91bac09bcff691f9372e018bde744e9e28a5f84b3a1812111dc not found: ID does not exist"
Jul 06 20:54:03.446922 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.446900    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"]
Jul 06 20:54:03.451247 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.451230    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp"]
Jul 06 20:54:03.472951 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:03.472932    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="453f51b1-e436-4038-bf28-4b129b2ae05e" path="/var/lib/kubelet/pods/453f51b1-e436-4038-bf28-4b129b2ae05e/volumes"
Jul 06 20:54:24.131793 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.131716    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"]
Jul 06 20:54:24.132264 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.132090    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453f51b1-e436-4038-bf28-4b129b2ae05e" containerName="storage-initializer"
Jul 06 20:54:24.132264 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.132107    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="453f51b1-e436-4038-bf28-4b129b2ae05e" containerName="storage-initializer"
Jul 06 20:54:24.132264 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.132121    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453f51b1-e436-4038-bf28-4b129b2ae05e" containerName="main"
Jul 06 20:54:24.132264 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.132127    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="453f51b1-e436-4038-bf28-4b129b2ae05e" containerName="main"
Jul 06 20:54:24.132264 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.132184    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="453f51b1-e436-4038-bf28-4b129b2ae05e" containerName="main"
Jul 06 20:54:24.135353 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.135338    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.138882 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.138860    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 06 20:54:24.138882 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.138878    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-6bj5g\""
Jul 06 20:54:24.144627 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.144607    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"]
Jul 06 20:54:24.221799 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.221763    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5f233277-d85b-4bc3-99ea-ba20e8902932-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.221799 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.221797    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdzh\" (UniqueName: \"kubernetes.io/projected/5f233277-d85b-4bc3-99ea-ba20e8902932-kube-api-access-9kdzh\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.222012 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.221885    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f233277-d85b-4bc3-99ea-ba20e8902932-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.323052 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.323024    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5f233277-d85b-4bc3-99ea-ba20e8902932-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.323191 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.323055    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdzh\" (UniqueName: \"kubernetes.io/projected/5f233277-d85b-4bc3-99ea-ba20e8902932-kube-api-access-9kdzh\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.323191 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.323078    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f233277-d85b-4bc3-99ea-ba20e8902932-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.323393 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.323375    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5f233277-d85b-4bc3-99ea-ba20e8902932-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.325426 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.325402    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f233277-d85b-4bc3-99ea-ba20e8902932-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.334164 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.334139    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdzh\" (UniqueName: \"kubernetes.io/projected/5f233277-d85b-4bc3-99ea-ba20e8902932-kube-api-access-9kdzh\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.444642 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.444550    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:24.573338 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:24.573316    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"]
Jul 06 20:54:24.575272 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:54:24.575243    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f233277_d85b_4bc3_99ea_ba20e8902932.slice/crio-ee2b846fa6d37f6861a4e952e403b099fd706fd83932d501b98bc0779f8269b7 WatchSource:0}: Error finding container ee2b846fa6d37f6861a4e952e403b099fd706fd83932d501b98bc0779f8269b7: Status 404 returned error can't find the container with id ee2b846fa6d37f6861a4e952e403b099fd706fd83932d501b98bc0779f8269b7
Jul 06 20:54:25.524168 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:25.524135    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s" event={"ID":"5f233277-d85b-4bc3-99ea-ba20e8902932","Type":"ContainerStarted","Data":"a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475"}
Jul 06 20:54:25.524577 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:25.524179    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s" event={"ID":"5f233277-d85b-4bc3-99ea-ba20e8902932","Type":"ContainerStarted","Data":"ee2b846fa6d37f6861a4e952e403b099fd706fd83932d501b98bc0779f8269b7"}
Jul 06 20:54:25.524577 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:25.524277    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:54:25.542692 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:25.542647    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s" podStartSLOduration=1.542634285 podStartE2EDuration="1.542634285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:54:25.541404921 +0000 UTC m=+1178.482708768" watchObservedRunningTime="2026-07-06 20:54:25.542634285 +0000 UTC m=+1178.483938132"
Jul 06 20:54:50.230369 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.230339    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"]
Jul 06 20:54:50.234670 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.234647    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.237321 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.237298    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:54:50.238317 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.238296    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 06 20:54:50.238403 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.238296    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-z6pqq\""
Jul 06 20:54:50.245947 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.245929    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"]
Jul 06 20:54:50.349848 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.349804    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.349998 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.349865    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zrrg\" (UniqueName: \"kubernetes.io/projected/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kube-api-access-6zrrg\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.349998 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.349922    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.349998 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.349942    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.456392 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.456363    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.456556 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.456411    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zrrg\" (UniqueName: \"kubernetes.io/projected/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kube-api-access-6zrrg\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.456556 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.456501    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.456556 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.456535    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.456948 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.456927    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.457006 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.456936    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.459182 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.459156    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.465508 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.465485    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zrrg\" (UniqueName: \"kubernetes.io/projected/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kube-api-access-6zrrg\") pod \"auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.543952 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.543893    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:50.679883 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:50.679851    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"]
Jul 06 20:54:50.681360 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:54:50.681336    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457e0845_e7e4_48eb_9c9a_cdb61b8d3814.slice/crio-a89103e2c7f7003c38832db4b21932e3156af0acedcdc97e80121bf99b6c82b1 WatchSource:0}: Error finding container a89103e2c7f7003c38832db4b21932e3156af0acedcdc97e80121bf99b6c82b1: Status 404 returned error can't find the container with id a89103e2c7f7003c38832db4b21932e3156af0acedcdc97e80121bf99b6c82b1
Jul 06 20:54:51.639110 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:51.639016    2550 generic.go:356] "Generic (PLEG): container finished" podID="457e0845-e7e4-48eb-9c9a-cdb61b8d3814" containerID="9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0" exitCode=0
Jul 06 20:54:51.639110 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:51.639072    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l" event={"ID":"457e0845-e7e4-48eb-9c9a-cdb61b8d3814","Type":"ContainerDied","Data":"9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0"}
Jul 06 20:54:51.639568 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:51.639115    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l" event={"ID":"457e0845-e7e4-48eb-9c9a-cdb61b8d3814","Type":"ContainerStarted","Data":"a89103e2c7f7003c38832db4b21932e3156af0acedcdc97e80121bf99b6c82b1"}
Jul 06 20:54:52.645826 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:52.645774    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l" event={"ID":"457e0845-e7e4-48eb-9c9a-cdb61b8d3814","Type":"ContainerStarted","Data":"9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e"}
Jul 06 20:54:52.646263 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:52.645855    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:54:52.666603 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:52.666551    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l" podStartSLOduration=2.033259056 podStartE2EDuration="2.66653681s" totalImagesPullingTime="0s" totalInitContainerRuntime="633.277754ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:54:52.665297563 +0000 UTC m=+1205.606601421" watchObservedRunningTime="2026-07-06 20:54:52.66653681 +0000 UTC m=+1205.607840656"
Jul 06 20:54:56.530092 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:54:56.530056    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:55:23.652844 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:23.652803    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:55:45.082359 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.082269    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"]
Jul 06 20:55:45.091116 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:55:45.091073    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 20:55:45.091116 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:55:45.091116 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:55:45.091116 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475" cmd=["/bin/sleep","15"]
Jul 06 20:55:45.091416 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.091134    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s" message="" err=<
Jul 06 20:55:45.091416 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:55:45.091416 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:55:45.091416 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 20:55:45.091416 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:55:45.091152    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 20:55:45.091416 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:55:45.091416 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:55:45.091416 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s" podUID="5f233277-d85b-4bc3-99ea-ba20e8902932" containerName="main" containerID="cri-o://a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475"
Jul 06 20:55:45.091416 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.091196    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s" podUID="5f233277-d85b-4bc3-99ea-ba20e8902932" containerName="main" containerID="cri-o://a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475" gracePeriod=60
Jul 06 20:55:45.331620 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.331598    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:55:45.429997 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.429918    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f233277-d85b-4bc3-99ea-ba20e8902932-tls-certs\") pod \"5f233277-d85b-4bc3-99ea-ba20e8902932\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") "
Jul 06 20:55:45.429997 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.429993    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5f233277-d85b-4bc3-99ea-ba20e8902932-tokenizer-uds\") pod \"5f233277-d85b-4bc3-99ea-ba20e8902932\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") "
Jul 06 20:55:45.430171 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.430029    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kdzh\" (UniqueName: \"kubernetes.io/projected/5f233277-d85b-4bc3-99ea-ba20e8902932-kube-api-access-9kdzh\") pod \"5f233277-d85b-4bc3-99ea-ba20e8902932\" (UID: \"5f233277-d85b-4bc3-99ea-ba20e8902932\") "
Jul 06 20:55:45.430263 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.430242    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f233277-d85b-4bc3-99ea-ba20e8902932-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5f233277-d85b-4bc3-99ea-ba20e8902932" (UID: "5f233277-d85b-4bc3-99ea-ba20e8902932"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:55:45.432016 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.431997    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f233277-d85b-4bc3-99ea-ba20e8902932-kube-api-access-9kdzh" (OuterVolumeSpecName: "kube-api-access-9kdzh") pod "5f233277-d85b-4bc3-99ea-ba20e8902932" (UID: "5f233277-d85b-4bc3-99ea-ba20e8902932"). InnerVolumeSpecName "kube-api-access-9kdzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:55:45.432074 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.432030    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f233277-d85b-4bc3-99ea-ba20e8902932-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f233277-d85b-4bc3-99ea-ba20e8902932" (UID: "5f233277-d85b-4bc3-99ea-ba20e8902932"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:55:45.531587 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.531561    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5f233277-d85b-4bc3-99ea-ba20e8902932-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:55:45.531587 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.531584    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kdzh\" (UniqueName: \"kubernetes.io/projected/5f233277-d85b-4bc3-99ea-ba20e8902932-kube-api-access-9kdzh\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:55:45.531738 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.531594    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f233277-d85b-4bc3-99ea-ba20e8902932-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:55:45.884354 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.884321    2550 generic.go:356] "Generic (PLEG): container finished" podID="5f233277-d85b-4bc3-99ea-ba20e8902932" containerID="a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475" exitCode=0
Jul 06 20:55:45.884521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.884369    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s" event={"ID":"5f233277-d85b-4bc3-99ea-ba20e8902932","Type":"ContainerDied","Data":"a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475"}
Jul 06 20:55:45.884521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.884376    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"
Jul 06 20:55:45.884521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.884407    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s" event={"ID":"5f233277-d85b-4bc3-99ea-ba20e8902932","Type":"ContainerDied","Data":"ee2b846fa6d37f6861a4e952e403b099fd706fd83932d501b98bc0779f8269b7"}
Jul 06 20:55:45.884521 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.884429    2550 scope.go:117] "RemoveContainer" containerID="a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475"
Jul 06 20:55:45.892917 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.892892    2550 scope.go:117] "RemoveContainer" containerID="a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475"
Jul 06 20:55:45.893168 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:55:45.893149    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475\": container with ID starting with a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475 not found: ID does not exist" containerID="a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475"
Jul 06 20:55:45.893229 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.893175    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475"} err="failed to get container status \"a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475\": rpc error: code = NotFound desc = could not find container \"a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475\": container with ID starting with a19c78d7e39d882b4043ecdea367aae7650d33f235c3128ef69a031b6ba2b475 not found: ID does not exist"
Jul 06 20:55:45.901437 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.901418    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"]
Jul 06 20:55:45.905176 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:45.905156    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s"]
Jul 06 20:55:47.473886 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:55:47.473852    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f233277-d85b-4bc3-99ea-ba20e8902932" path="/var/lib/kubelet/pods/5f233277-d85b-4bc3-99ea-ba20e8902932/volumes"
Jul 06 20:56:06.106250 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:06.106211    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"]
Jul 06 20:56:21.126571 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.126509    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9" podUID="62a7b61f-8ad8-4240-aaa7-08de4ac64a15" containerName="main" containerID="cri-o://02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9" gracePeriod=45
Jul 06 20:56:21.369958 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.369934    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:56:21.433313 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.433239    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-model-cache\") pod \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") "
Jul 06 20:56:21.433313 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.433269    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-dshm\") pod \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") "
Jul 06 20:56:21.433313 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.433300    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x7zp\" (UniqueName: \"kubernetes.io/projected/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-kube-api-access-6x7zp\") pod \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") "
Jul 06 20:56:21.433570 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.433364    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tmp-dir\") pod \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") "
Jul 06 20:56:21.433570 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.433392    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tls-certs\") pod \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") "
Jul 06 20:56:21.433570 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.433427    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-home\") pod \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\" (UID: \"62a7b61f-8ad8-4240-aaa7-08de4ac64a15\") "
Jul 06 20:56:21.433570 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.433505    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-model-cache" (OuterVolumeSpecName: "model-cache") pod "62a7b61f-8ad8-4240-aaa7-08de4ac64a15" (UID: "62a7b61f-8ad8-4240-aaa7-08de4ac64a15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:21.433777 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.433656    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62a7b61f-8ad8-4240-aaa7-08de4ac64a15" (UID: "62a7b61f-8ad8-4240-aaa7-08de4ac64a15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:21.433777 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.433724    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-home" (OuterVolumeSpecName: "home") pod "62a7b61f-8ad8-4240-aaa7-08de4ac64a15" (UID: "62a7b61f-8ad8-4240-aaa7-08de4ac64a15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:21.433777 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.433759    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:56:21.435239 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.435213    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-dshm" (OuterVolumeSpecName: "dshm") pod "62a7b61f-8ad8-4240-aaa7-08de4ac64a15" (UID: "62a7b61f-8ad8-4240-aaa7-08de4ac64a15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:21.435375 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.435354    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62a7b61f-8ad8-4240-aaa7-08de4ac64a15" (UID: "62a7b61f-8ad8-4240-aaa7-08de4ac64a15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:56:21.435375 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.435365    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-kube-api-access-6x7zp" (OuterVolumeSpecName: "kube-api-access-6x7zp") pod "62a7b61f-8ad8-4240-aaa7-08de4ac64a15" (UID: "62a7b61f-8ad8-4240-aaa7-08de4ac64a15"). InnerVolumeSpecName "kube-api-access-6x7zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:56:21.534406 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.534383    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:56:21.534406 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.534403    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:56:21.534406 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.534411    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:56:21.534585 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.534419    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x7zp\" (UniqueName: \"kubernetes.io/projected/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-kube-api-access-6x7zp\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:56:21.534585 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:21.534429    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7b61f-8ad8-4240-aaa7-08de4ac64a15-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:56:22.042855 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:22.042793    2550 generic.go:356] "Generic (PLEG): container finished" podID="62a7b61f-8ad8-4240-aaa7-08de4ac64a15" containerID="02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9" exitCode=0
Jul 06 20:56:22.042855 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:22.042859    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"
Jul 06 20:56:22.043113 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:22.042875    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9" event={"ID":"62a7b61f-8ad8-4240-aaa7-08de4ac64a15","Type":"ContainerDied","Data":"02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9"}
Jul 06 20:56:22.043113 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:22.042929    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9" event={"ID":"62a7b61f-8ad8-4240-aaa7-08de4ac64a15","Type":"ContainerDied","Data":"fd7f5464412ad09527d1d2cb6a24531b3f50f8526a87204b940be224a17953b9"}
Jul 06 20:56:22.043113 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:22.042952    2550 scope.go:117] "RemoveContainer" containerID="02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9"
Jul 06 20:56:22.051153 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:22.051138    2550 scope.go:117] "RemoveContainer" containerID="02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9"
Jul 06 20:56:22.051421 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:56:22.051403    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9\": container with ID starting with 02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9 not found: ID does not exist" containerID="02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9"
Jul 06 20:56:22.051499 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:22.051428    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9"} err="failed to get container status \"02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9\": rpc error: code = NotFound desc = could not find container \"02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9\": container with ID starting with 02e2186d95719cbda733d23bfafeb7990860ce0cca8113438aaae22c59ca08a9 not found: ID does not exist"
Jul 06 20:56:22.059912 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:22.059890    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"]
Jul 06 20:56:22.065597 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:22.065578    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9"]
Jul 06 20:56:23.473108 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:23.473075    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a7b61f-8ad8-4240-aaa7-08de4ac64a15" path="/var/lib/kubelet/pods/62a7b61f-8ad8-4240-aaa7-08de4ac64a15/volumes"
Jul 06 20:56:50.682888 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.682856    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"]
Jul 06 20:56:50.683248 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.683179    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f233277-d85b-4bc3-99ea-ba20e8902932" containerName="main"
Jul 06 20:56:50.683248 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.683191    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f233277-d85b-4bc3-99ea-ba20e8902932" containerName="main"
Jul 06 20:56:50.683248 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.683223    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a7b61f-8ad8-4240-aaa7-08de4ac64a15" containerName="main"
Jul 06 20:56:50.683248 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.683231    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a7b61f-8ad8-4240-aaa7-08de4ac64a15" containerName="main"
Jul 06 20:56:50.683375 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.683280    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a7b61f-8ad8-4240-aaa7-08de4ac64a15" containerName="main"
Jul 06 20:56:50.683375 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.683291    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f233277-d85b-4bc3-99ea-ba20e8902932" containerName="main"
Jul 06 20:56:50.687314 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.687293    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.689891 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.689866    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 06 20:56:50.690032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.690018    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-zstl9\""
Jul 06 20:56:50.699111 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.699089    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"]
Jul 06 20:56:50.872264 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.872230    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39db9fb1-b891-480c-bde3-35429cf9ed37-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.872449 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.872278    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.872449 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.872302    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.872449 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.872404    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvbxq\" (UniqueName: \"kubernetes.io/projected/39db9fb1-b891-480c-bde3-35429cf9ed37-kube-api-access-mvbxq\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.972926 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.972828    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39db9fb1-b891-480c-bde3-35429cf9ed37-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.972926 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.972888    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.972926 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.972914    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.973212 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.972972    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbxq\" (UniqueName: \"kubernetes.io/projected/39db9fb1-b891-480c-bde3-35429cf9ed37-kube-api-access-mvbxq\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.973377 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.973344    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.973377 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.973369    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.975377 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.975358    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39db9fb1-b891-480c-bde3-35429cf9ed37-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.984336 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.984309    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbxq\" (UniqueName: \"kubernetes.io/projected/39db9fb1-b891-480c-bde3-35429cf9ed37-kube-api-access-mvbxq\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:50.997158 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:50.997139    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:51.139396 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:51.139371    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"]
Jul 06 20:56:51.142054 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:56:51.142023    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39db9fb1_b891_480c_bde3_35429cf9ed37.slice/crio-571f17bf792bcb9ceba1bfe0d50a89d953ed2fa6e83fbe5970093d63c9832bf5 WatchSource:0}: Error finding container 571f17bf792bcb9ceba1bfe0d50a89d953ed2fa6e83fbe5970093d63c9832bf5: Status 404 returned error can't find the container with id 571f17bf792bcb9ceba1bfe0d50a89d953ed2fa6e83fbe5970093d63c9832bf5
Jul 06 20:56:51.175431 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:51.175403    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t" event={"ID":"39db9fb1-b891-480c-bde3-35429cf9ed37","Type":"ContainerStarted","Data":"571f17bf792bcb9ceba1bfe0d50a89d953ed2fa6e83fbe5970093d63c9832bf5"}
Jul 06 20:56:52.181560 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:52.181457    2550 generic.go:356] "Generic (PLEG): container finished" podID="39db9fb1-b891-480c-bde3-35429cf9ed37" containerID="119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137" exitCode=0
Jul 06 20:56:52.182243 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:52.181547    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t" event={"ID":"39db9fb1-b891-480c-bde3-35429cf9ed37","Type":"ContainerDied","Data":"119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137"}
Jul 06 20:56:53.188658 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:53.188621    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t" event={"ID":"39db9fb1-b891-480c-bde3-35429cf9ed37","Type":"ContainerStarted","Data":"bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b"}
Jul 06 20:56:53.189110 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:53.188716    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:56:53.211467 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:56:53.211418    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t" podStartSLOduration=2.525378318 podStartE2EDuration="3.211402576s" totalImagesPullingTime="0s" totalInitContainerRuntime="686.024258ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:56:53.209325302 +0000 UTC m=+1326.150629208" watchObservedRunningTime="2026-07-06 20:56:53.211402576 +0000 UTC m=+1326.152706422"
Jul 06 20:57:05.247819 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.247773    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"]
Jul 06 20:57:05.256517 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:57:05.256481    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 20:57:05.256517 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:57:05.256517 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:57:05.256517 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e" cmd=["/bin/sleep","15"]
Jul 06 20:57:05.256763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.256531    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l" message="" err=<
Jul 06 20:57:05.256763 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:57:05.256763 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:57:05.256763 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 20:57:05.256763 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:57:05.256543    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 20:57:05.256763 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:57:05.256763 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:57:05.256763 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l" podUID="457e0845-e7e4-48eb-9c9a-cdb61b8d3814" containerName="main" containerID="cri-o://9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e"
Jul 06 20:57:05.256763 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.256586    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l" podUID="457e0845-e7e4-48eb-9c9a-cdb61b8d3814" containerName="main" containerID="cri-o://9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e" gracePeriod=60
Jul 06 20:57:05.501446 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.501391    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:57:05.594757 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.594718    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zrrg\" (UniqueName: \"kubernetes.io/projected/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kube-api-access-6zrrg\") pod \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") "
Jul 06 20:57:05.594757 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.594761    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tls-certs\") pod \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") "
Jul 06 20:57:05.595032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.594825    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kserve-provision-location\") pod \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") "
Jul 06 20:57:05.595032 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.594892    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tokenizer-uds\") pod \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\" (UID: \"457e0845-e7e4-48eb-9c9a-cdb61b8d3814\") "
Jul 06 20:57:05.595194 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.595168    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "457e0845-e7e4-48eb-9c9a-cdb61b8d3814" (UID: "457e0845-e7e4-48eb-9c9a-cdb61b8d3814"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:57:05.595529 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.595507    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "457e0845-e7e4-48eb-9c9a-cdb61b8d3814" (UID: "457e0845-e7e4-48eb-9c9a-cdb61b8d3814"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:57:05.596899 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.596880    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "457e0845-e7e4-48eb-9c9a-cdb61b8d3814" (UID: "457e0845-e7e4-48eb-9c9a-cdb61b8d3814"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:57:05.596992 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.596933    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kube-api-access-6zrrg" (OuterVolumeSpecName: "kube-api-access-6zrrg") pod "457e0845-e7e4-48eb-9c9a-cdb61b8d3814" (UID: "457e0845-e7e4-48eb-9c9a-cdb61b8d3814"). InnerVolumeSpecName "kube-api-access-6zrrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:57:05.696531 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.696493    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:57:05.696531 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.696524    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:57:05.696531 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.696534    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zrrg\" (UniqueName: \"kubernetes.io/projected/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-kube-api-access-6zrrg\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:57:05.696531 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:05.696543    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457e0845-e7e4-48eb-9c9a-cdb61b8d3814-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:57:06.249370 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.249336    2550 generic.go:356] "Generic (PLEG): container finished" podID="457e0845-e7e4-48eb-9c9a-cdb61b8d3814" containerID="9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e" exitCode=0
Jul 06 20:57:06.249796 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.249399    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l" event={"ID":"457e0845-e7e4-48eb-9c9a-cdb61b8d3814","Type":"ContainerDied","Data":"9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e"}
Jul 06 20:57:06.249796 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.249441    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l" event={"ID":"457e0845-e7e4-48eb-9c9a-cdb61b8d3814","Type":"ContainerDied","Data":"a89103e2c7f7003c38832db4b21932e3156af0acedcdc97e80121bf99b6c82b1"}
Jul 06 20:57:06.249796 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.249461    2550 scope.go:117] "RemoveContainer" containerID="9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e"
Jul 06 20:57:06.249796 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.249403    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"
Jul 06 20:57:06.258715 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.258556    2550 scope.go:117] "RemoveContainer" containerID="9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0"
Jul 06 20:57:06.265754 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.265739    2550 scope.go:117] "RemoveContainer" containerID="9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e"
Jul 06 20:57:06.265995 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:57:06.265973    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e\": container with ID starting with 9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e not found: ID does not exist" containerID="9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e"
Jul 06 20:57:06.266071 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.266007    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e"} err="failed to get container status \"9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e\": rpc error: code = NotFound desc = could not find container \"9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e\": container with ID starting with 9249703933dd0e17720c6fb9b8237b945e4d8c81ef34fac49117429b5783e20e not found: ID does not exist"
Jul 06 20:57:06.266071 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.266030    2550 scope.go:117] "RemoveContainer" containerID="9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0"
Jul 06 20:57:06.266284 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:57:06.266267    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0\": container with ID starting with 9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0 not found: ID does not exist" containerID="9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0"
Jul 06 20:57:06.266326 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.266291    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0"} err="failed to get container status \"9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0\": rpc error: code = NotFound desc = could not find container \"9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0\": container with ID starting with 9b3c5407e9126e190809a7b2ef6c77864ee2999362b94e21e98be6331e880af0 not found: ID does not exist"
Jul 06 20:57:06.271057 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.271037    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"]
Jul 06 20:57:06.274755 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:06.274736    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l"]
Jul 06 20:57:07.474088 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:07.474057    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="457e0845-e7e4-48eb-9c9a-cdb61b8d3814" path="/var/lib/kubelet/pods/457e0845-e7e4-48eb-9c9a-cdb61b8d3814/volumes"
Jul 06 20:57:24.194988 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:57:24.194960    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:58:00.965135 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:00.965098    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"]
Jul 06 20:58:00.965692 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:00.965442    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457e0845-e7e4-48eb-9c9a-cdb61b8d3814" containerName="main"
Jul 06 20:58:00.965692 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:00.965453    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="457e0845-e7e4-48eb-9c9a-cdb61b8d3814" containerName="main"
Jul 06 20:58:00.965692 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:00.965465    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457e0845-e7e4-48eb-9c9a-cdb61b8d3814" containerName="storage-initializer"
Jul 06 20:58:00.965692 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:00.965471    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="457e0845-e7e4-48eb-9c9a-cdb61b8d3814" containerName="storage-initializer"
Jul 06 20:58:00.965692 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:00.965536    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="457e0845-e7e4-48eb-9c9a-cdb61b8d3814" containerName="main"
Jul 06 20:58:00.969662 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:00.969646    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:00.972062 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:00.972040    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Jul 06 20:58:00.977892 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:00.977870    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"]
Jul 06 20:58:01.055937 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.055906    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.056083 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.055954    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.056083 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.055976    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.056083 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.056040    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69pg2\" (UniqueName: \"kubernetes.io/projected/7a2f28d5-b884-468d-af2f-72feabc4c7bf-kube-api-access-69pg2\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.056083 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.056071    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.056218 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.056094    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.156583 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.156550    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.156738 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.156616    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.156738 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.156636    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.156738 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.156656    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69pg2\" (UniqueName: \"kubernetes.io/projected/7a2f28d5-b884-468d-af2f-72feabc4c7bf-kube-api-access-69pg2\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.156738 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.156675    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.156977 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.156762    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.157045 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.156976    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.157045 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.156996    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.157168 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.157065    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.158841 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.158798    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.159175 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.159157    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.164599 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.164580    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69pg2\" (UniqueName: \"kubernetes.io/projected/7a2f28d5-b884-468d-af2f-72feabc4c7bf-kube-api-access-69pg2\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.280784 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.280713    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:01.467516 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.467485    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"]
Jul 06 20:58:01.468848 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:58:01.468823    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2f28d5_b884_468d_af2f_72feabc4c7bf.slice/crio-9da1fbf8aab2804f98180fceb3a5d281651ca7532e61280a8cee49236d098ce0 WatchSource:0}: Error finding container 9da1fbf8aab2804f98180fceb3a5d281651ca7532e61280a8cee49236d098ce0: Status 404 returned error can't find the container with id 9da1fbf8aab2804f98180fceb3a5d281651ca7532e61280a8cee49236d098ce0
Jul 06 20:58:01.490776 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:01.490749    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv" event={"ID":"7a2f28d5-b884-468d-af2f-72feabc4c7bf","Type":"ContainerStarted","Data":"9da1fbf8aab2804f98180fceb3a5d281651ca7532e61280a8cee49236d098ce0"}
Jul 06 20:58:02.497384 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:02.497343    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv" event={"ID":"7a2f28d5-b884-468d-af2f-72feabc4c7bf","Type":"ContainerStarted","Data":"9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063"}
Jul 06 20:58:02.517367 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:02.517320    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv" podStartSLOduration=2.517307256 podStartE2EDuration="2.517307256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:02.516330972 +0000 UTC m=+1395.457634833" watchObservedRunningTime="2026-07-06 20:58:02.517307256 +0000 UTC m=+1395.458611144"
Jul 06 20:58:03.281189 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:03.281159    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:11.280849 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:11.280800    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:11.297876 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:11.297854    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:11.545711 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:11.545620    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 20:58:50.199239 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.199208    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"]
Jul 06 20:58:50.207861 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:58:50.207827    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 20:58:50.207861 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:58:50.207861 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:58:50.207861 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b" cmd=["/bin/sleep","15"]
Jul 06 20:58:50.208142 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.207875    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t" message="" err=<
Jul 06 20:58:50.208142 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:58:50.208142 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:58:50.208142 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 20:58:50.208142 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:58:50.207887    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 20:58:50.208142 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:58:50.208142 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 20:58:50.208142 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t" podUID="39db9fb1-b891-480c-bde3-35429cf9ed37" containerName="main" containerID="cri-o://bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b"
Jul 06 20:58:50.208142 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.207926    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t" podUID="39db9fb1-b891-480c-bde3-35429cf9ed37" containerName="main" containerID="cri-o://bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b" gracePeriod=60
Jul 06 20:58:50.453533 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.453473    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:58:50.569033 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.568999    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-tokenizer-uds\") pod \"39db9fb1-b891-480c-bde3-35429cf9ed37\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") "
Jul 06 20:58:50.569181 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.569038    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvbxq\" (UniqueName: \"kubernetes.io/projected/39db9fb1-b891-480c-bde3-35429cf9ed37-kube-api-access-mvbxq\") pod \"39db9fb1-b891-480c-bde3-35429cf9ed37\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") "
Jul 06 20:58:50.569181 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.569091    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39db9fb1-b891-480c-bde3-35429cf9ed37-tls-certs\") pod \"39db9fb1-b891-480c-bde3-35429cf9ed37\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") "
Jul 06 20:58:50.569181 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.569112    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-kserve-provision-location\") pod \"39db9fb1-b891-480c-bde3-35429cf9ed37\" (UID: \"39db9fb1-b891-480c-bde3-35429cf9ed37\") "
Jul 06 20:58:50.569285 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.569239    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "39db9fb1-b891-480c-bde3-35429cf9ed37" (UID: "39db9fb1-b891-480c-bde3-35429cf9ed37"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:50.569348 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.569334    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:58:50.569850 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.569802    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39db9fb1-b891-480c-bde3-35429cf9ed37" (UID: "39db9fb1-b891-480c-bde3-35429cf9ed37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:50.571146 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.571127    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39db9fb1-b891-480c-bde3-35429cf9ed37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39db9fb1-b891-480c-bde3-35429cf9ed37" (UID: "39db9fb1-b891-480c-bde3-35429cf9ed37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:58:50.571248 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.571233    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39db9fb1-b891-480c-bde3-35429cf9ed37-kube-api-access-mvbxq" (OuterVolumeSpecName: "kube-api-access-mvbxq") pod "39db9fb1-b891-480c-bde3-35429cf9ed37" (UID: "39db9fb1-b891-480c-bde3-35429cf9ed37"). InnerVolumeSpecName "kube-api-access-mvbxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:58:50.670396 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.670375    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvbxq\" (UniqueName: \"kubernetes.io/projected/39db9fb1-b891-480c-bde3-35429cf9ed37-kube-api-access-mvbxq\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:58:50.670396 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.670396    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39db9fb1-b891-480c-bde3-35429cf9ed37-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:58:50.670524 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.670407    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39db9fb1-b891-480c-bde3-35429cf9ed37-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 20:58:50.705440 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.705379    2550 generic.go:356] "Generic (PLEG): container finished" podID="39db9fb1-b891-480c-bde3-35429cf9ed37" containerID="bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b" exitCode=0
Jul 06 20:58:50.705440 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.705428    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t" event={"ID":"39db9fb1-b891-480c-bde3-35429cf9ed37","Type":"ContainerDied","Data":"bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b"}
Jul 06 20:58:50.705440 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.705432    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"
Jul 06 20:58:50.705590 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.705467    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t" event={"ID":"39db9fb1-b891-480c-bde3-35429cf9ed37","Type":"ContainerDied","Data":"571f17bf792bcb9ceba1bfe0d50a89d953ed2fa6e83fbe5970093d63c9832bf5"}
Jul 06 20:58:50.705590 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.705486    2550 scope.go:117] "RemoveContainer" containerID="bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b"
Jul 06 20:58:50.714722 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.714705    2550 scope.go:117] "RemoveContainer" containerID="119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137"
Jul 06 20:58:50.722584 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.722568    2550 scope.go:117] "RemoveContainer" containerID="bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b"
Jul 06 20:58:50.722869 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:58:50.722844    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b\": container with ID starting with bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b not found: ID does not exist" containerID="bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b"
Jul 06 20:58:50.722972 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.722872    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b"} err="failed to get container status \"bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b\": rpc error: code = NotFound desc = could not find container \"bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b\": container with ID starting with bfdb9dfaddabed6740b9cf2747d860f733bd8ef838d0178e1567a6f57ea2cf2b not found: ID does not exist"
Jul 06 20:58:50.722972 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.722907    2550 scope.go:117] "RemoveContainer" containerID="119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137"
Jul 06 20:58:50.723142 ip-10-0-131-45 kubenswrapper[2550]: E0706 20:58:50.723125    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137\": container with ID starting with 119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137 not found: ID does not exist" containerID="119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137"
Jul 06 20:58:50.723190 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.723148    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137"} err="failed to get container status \"119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137\": rpc error: code = NotFound desc = could not find container \"119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137\": container with ID starting with 119f7bbf815e5cadb666567962ad6cb926072dd75f47d6308af6502804961137 not found: ID does not exist"
Jul 06 20:58:50.729492 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.729471    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"]
Jul 06 20:58:50.732929 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:50.732910    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t"]
Jul 06 20:58:51.473737 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:58:51.473697    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39db9fb1-b891-480c-bde3-35429cf9ed37" path="/var/lib/kubelet/pods/39db9fb1-b891-480c-bde3-35429cf9ed37/volumes"
Jul 06 20:59:32.177116 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.177084    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"]
Jul 06 20:59:32.177511 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.177402    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39db9fb1-b891-480c-bde3-35429cf9ed37" containerName="main"
Jul 06 20:59:32.177511 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.177414    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="39db9fb1-b891-480c-bde3-35429cf9ed37" containerName="main"
Jul 06 20:59:32.177511 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.177436    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39db9fb1-b891-480c-bde3-35429cf9ed37" containerName="storage-initializer"
Jul 06 20:59:32.177511 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.177442    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="39db9fb1-b891-480c-bde3-35429cf9ed37" containerName="storage-initializer"
Jul 06 20:59:32.177511 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.177501    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="39db9fb1-b891-480c-bde3-35429cf9ed37" containerName="main"
Jul 06 20:59:32.180551 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.180536    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.183270 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.183248    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:59:32.183390 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.183277    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-wchbq\""
Jul 06 20:59:32.183390 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.183276    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 06 20:59:32.191225 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.191206    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"]
Jul 06 20:59:32.221944 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.221916    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbxcd\" (UniqueName: \"kubernetes.io/projected/10010295-efc7-4896-92c0-8f6d27a67934-kube-api-access-nbxcd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.222075 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.221964    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.222075 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.222021    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.222075 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.222059    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10010295-efc7-4896-92c0-8f6d27a67934-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.322665 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.322632    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.322856 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.322690    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.322856 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.322746    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10010295-efc7-4896-92c0-8f6d27a67934-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.322856 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.322783    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbxcd\" (UniqueName: \"kubernetes.io/projected/10010295-efc7-4896-92c0-8f6d27a67934-kube-api-access-nbxcd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.323050 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.323029    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.323105 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.323058    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.325246 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.325224    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10010295-efc7-4896-92c0-8f6d27a67934-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.330693 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.330673    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbxcd\" (UniqueName: \"kubernetes.io/projected/10010295-efc7-4896-92c0-8f6d27a67934-kube-api-access-nbxcd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.489729 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.489640    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:32.626056 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.626028    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"]
Jul 06 20:59:32.629505 ip-10-0-131-45 kubenswrapper[2550]: W0706 20:59:32.629475    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10010295_efc7_4896_92c0_8f6d27a67934.slice/crio-ebdb874c197da3af02879febc4fe4d773741cae24193089f68f3ab5ea4413e22 WatchSource:0}: Error finding container ebdb874c197da3af02879febc4fe4d773741cae24193089f68f3ab5ea4413e22: Status 404 returned error can't find the container with id ebdb874c197da3af02879febc4fe4d773741cae24193089f68f3ab5ea4413e22
Jul 06 20:59:32.887017 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.886980    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25" event={"ID":"10010295-efc7-4896-92c0-8f6d27a67934","Type":"ContainerStarted","Data":"8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a"}
Jul 06 20:59:32.887207 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:32.887031    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25" event={"ID":"10010295-efc7-4896-92c0-8f6d27a67934","Type":"ContainerStarted","Data":"ebdb874c197da3af02879febc4fe4d773741cae24193089f68f3ab5ea4413e22"}
Jul 06 20:59:33.892769 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:33.892738    2550 generic.go:356] "Generic (PLEG): container finished" podID="10010295-efc7-4896-92c0-8f6d27a67934" containerID="8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a" exitCode=0
Jul 06 20:59:33.893189 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:33.892784    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25" event={"ID":"10010295-efc7-4896-92c0-8f6d27a67934","Type":"ContainerDied","Data":"8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a"}
Jul 06 20:59:34.899105 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:34.899063    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25" event={"ID":"10010295-efc7-4896-92c0-8f6d27a67934","Type":"ContainerStarted","Data":"626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e"}
Jul 06 20:59:34.899497 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:34.899231    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 20:59:34.922256 ip-10-0-131-45 kubenswrapper[2550]: I0706 20:59:34.922211    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25" podStartSLOduration=2.278135858 podStartE2EDuration="2.922198958s" totalImagesPullingTime="0s" totalInitContainerRuntime="644.0631ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:34.91974842 +0000 UTC m=+1487.861052268" watchObservedRunningTime="2026-07-06 20:59:34.922198958 +0000 UTC m=+1487.863502836"
Jul 06 21:00:05.905054 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:00:05.905025    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 21:01:03.845934 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:03.845899    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"]
Jul 06 21:01:03.855104 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:01:03.855069    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:01:03.855104 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:01:03.855104 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:01:03.855104 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e" cmd=["/bin/sleep","15"]
Jul 06 21:01:03.855406 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:03.855123    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25" message="" err=<
Jul 06 21:01:03.855406 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:01:03.855406 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:01:03.855406 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:01:03.855406 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:01:03.855139    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:01:03.855406 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:01:03.855406 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:01:03.855406 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25" podUID="10010295-efc7-4896-92c0-8f6d27a67934" containerName="main" containerID="cri-o://626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e"
Jul 06 21:01:03.855406 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:03.855181    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25" podUID="10010295-efc7-4896-92c0-8f6d27a67934" containerName="main" containerID="cri-o://626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e" gracePeriod=60
Jul 06 21:01:04.100024 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.099968    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 21:01:04.198490 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.198460    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-kserve-provision-location\") pod \"10010295-efc7-4896-92c0-8f6d27a67934\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") "
Jul 06 21:01:04.198684 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.198508    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10010295-efc7-4896-92c0-8f6d27a67934-tls-certs\") pod \"10010295-efc7-4896-92c0-8f6d27a67934\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") "
Jul 06 21:01:04.198684 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.198581    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-tokenizer-uds\") pod \"10010295-efc7-4896-92c0-8f6d27a67934\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") "
Jul 06 21:01:04.198837 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.198714    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbxcd\" (UniqueName: \"kubernetes.io/projected/10010295-efc7-4896-92c0-8f6d27a67934-kube-api-access-nbxcd\") pod \"10010295-efc7-4896-92c0-8f6d27a67934\" (UID: \"10010295-efc7-4896-92c0-8f6d27a67934\") "
Jul 06 21:01:04.198907 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.198853    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "10010295-efc7-4896-92c0-8f6d27a67934" (UID: "10010295-efc7-4896-92c0-8f6d27a67934"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:01:04.199028 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.199013    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:01:04.199196 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.199178    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10010295-efc7-4896-92c0-8f6d27a67934" (UID: "10010295-efc7-4896-92c0-8f6d27a67934"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:01:04.200688 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.200668    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10010295-efc7-4896-92c0-8f6d27a67934-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10010295-efc7-4896-92c0-8f6d27a67934" (UID: "10010295-efc7-4896-92c0-8f6d27a67934"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:01:04.200768 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.200749    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10010295-efc7-4896-92c0-8f6d27a67934-kube-api-access-nbxcd" (OuterVolumeSpecName: "kube-api-access-nbxcd") pod "10010295-efc7-4896-92c0-8f6d27a67934" (UID: "10010295-efc7-4896-92c0-8f6d27a67934"). InnerVolumeSpecName "kube-api-access-nbxcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:01:04.288228 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.288198    2550 generic.go:356] "Generic (PLEG): container finished" podID="10010295-efc7-4896-92c0-8f6d27a67934" containerID="626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e" exitCode=0
Jul 06 21:01:04.288354 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.288247    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"
Jul 06 21:01:04.288354 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.288283    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25" event={"ID":"10010295-efc7-4896-92c0-8f6d27a67934","Type":"ContainerDied","Data":"626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e"}
Jul 06 21:01:04.288354 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.288333    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25" event={"ID":"10010295-efc7-4896-92c0-8f6d27a67934","Type":"ContainerDied","Data":"ebdb874c197da3af02879febc4fe4d773741cae24193089f68f3ab5ea4413e22"}
Jul 06 21:01:04.288354 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.288354    2550 scope.go:117] "RemoveContainer" containerID="626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e"
Jul 06 21:01:04.296658 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.296643    2550 scope.go:117] "RemoveContainer" containerID="8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a"
Jul 06 21:01:04.299968 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.299951    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10010295-efc7-4896-92c0-8f6d27a67934-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:01:04.300051 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.299969    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10010295-efc7-4896-92c0-8f6d27a67934-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:01:04.300051 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.299980    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbxcd\" (UniqueName: \"kubernetes.io/projected/10010295-efc7-4896-92c0-8f6d27a67934-kube-api-access-nbxcd\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:01:04.303841 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.303795    2550 scope.go:117] "RemoveContainer" containerID="626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e"
Jul 06 21:01:04.304057 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:01:04.304040    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e\": container with ID starting with 626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e not found: ID does not exist" containerID="626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e"
Jul 06 21:01:04.304119 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.304064    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e"} err="failed to get container status \"626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e\": rpc error: code = NotFound desc = could not find container \"626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e\": container with ID starting with 626a138a2bef5b8a9d06a042bbe2a59b71ae65d1b625b17eb9f4dd342d5c3f4e not found: ID does not exist"
Jul 06 21:01:04.304119 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.304079    2550 scope.go:117] "RemoveContainer" containerID="8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a"
Jul 06 21:01:04.304315 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:01:04.304297    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a\": container with ID starting with 8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a not found: ID does not exist" containerID="8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a"
Jul 06 21:01:04.304357 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.304324    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a"} err="failed to get container status \"8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a\": rpc error: code = NotFound desc = could not find container \"8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a\": container with ID starting with 8e2423f7cef0e6f8931c9083f7b39ed5251f7e35bf8be05ad084b7df70259a6a not found: ID does not exist"
Jul 06 21:01:04.312666 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.312644    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"]
Jul 06 21:01:04.316421 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:04.316401    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25"]
Jul 06 21:01:05.473422 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:01:05.473389    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10010295-efc7-4896-92c0-8f6d27a67934" path="/var/lib/kubelet/pods/10010295-efc7-4896-92c0-8f6d27a67934/volumes"
Jul 06 21:04:47.644167 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.644137    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"]
Jul 06 21:04:47.644553 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.644446    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10010295-efc7-4896-92c0-8f6d27a67934" containerName="main"
Jul 06 21:04:47.644553 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.644482    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="10010295-efc7-4896-92c0-8f6d27a67934" containerName="main"
Jul 06 21:04:47.644553 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.644506    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10010295-efc7-4896-92c0-8f6d27a67934" containerName="storage-initializer"
Jul 06 21:04:47.644553 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.644513    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="10010295-efc7-4896-92c0-8f6d27a67934" containerName="storage-initializer"
Jul 06 21:04:47.644680 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.644574    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="10010295-efc7-4896-92c0-8f6d27a67934" containerName="main"
Jul 06 21:04:47.647636 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.647616    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:47.651102 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.651082    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 06 21:04:47.651209 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.651082    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-jtxxv\""
Jul 06 21:04:47.655931 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.655905    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97883cf5-0738-4698-b483-05a36edf622c-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:47.656037 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.655935    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m484f\" (UniqueName: \"kubernetes.io/projected/97883cf5-0738-4698-b483-05a36edf622c-kube-api-access-m484f\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:47.656037 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.655955    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/97883cf5-0738-4698-b483-05a36edf622c-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:47.659305 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.659284    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"]
Jul 06 21:04:47.757109 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.757087    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97883cf5-0738-4698-b483-05a36edf622c-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:47.757228 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.757115    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m484f\" (UniqueName: \"kubernetes.io/projected/97883cf5-0738-4698-b483-05a36edf622c-kube-api-access-m484f\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:47.757228 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.757136    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/97883cf5-0738-4698-b483-05a36edf622c-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:47.757444 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.757430    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/97883cf5-0738-4698-b483-05a36edf622c-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:47.759466 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.759446    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97883cf5-0738-4698-b483-05a36edf622c-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:47.766274 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.766257    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m484f\" (UniqueName: \"kubernetes.io/projected/97883cf5-0738-4698-b483-05a36edf622c-kube-api-access-m484f\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:47.956681 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:47.956612    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:48.086679 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:48.086647    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"]
Jul 06 21:04:48.239923 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:48.239835    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt" event={"ID":"97883cf5-0738-4698-b483-05a36edf622c","Type":"ContainerStarted","Data":"c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b"}
Jul 06 21:04:48.239923 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:48.239887    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt" event={"ID":"97883cf5-0738-4698-b483-05a36edf622c","Type":"ContainerStarted","Data":"eac64ccc9c911351522beff50c986c2de2bd202343b177bc043465f0575ee8aa"}
Jul 06 21:04:48.239923 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:48.239914    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:04:48.259874 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:04:48.259831    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt" podStartSLOduration=1.259794164 podStartE2EDuration="1.259794164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:04:48.258364774 +0000 UTC m=+1801.199668646" watchObservedRunningTime="2026-07-06 21:04:48.259794164 +0000 UTC m=+1801.201098010"
Jul 06 21:05:15.831201 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:15.831170    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"]
Jul 06 21:05:19.245829 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:19.245782    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:05:30.850629 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:30.850586    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv" podUID="7a2f28d5-b884-468d-af2f-72feabc4c7bf" containerName="main" containerID="cri-o://9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063" gracePeriod=45
Jul 06 21:05:31.102009 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.101957    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 21:05:31.132980 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.132946    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69pg2\" (UniqueName: \"kubernetes.io/projected/7a2f28d5-b884-468d-af2f-72feabc4c7bf-kube-api-access-69pg2\") pod \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") "
Jul 06 21:05:31.133124 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.132986    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tmp-dir\") pod \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") "
Jul 06 21:05:31.133124 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.133009    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-dshm\") pod \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") "
Jul 06 21:05:31.133124 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.133042    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tls-certs\") pod \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") "
Jul 06 21:05:31.133124 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.133064    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-home\") pod \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") "
Jul 06 21:05:31.133124 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.133117    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-model-cache\") pod \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\" (UID: \"7a2f28d5-b884-468d-af2f-72feabc4c7bf\") "
Jul 06 21:05:31.133396 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.133261    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a2f28d5-b884-468d-af2f-72feabc4c7bf" (UID: "7a2f28d5-b884-468d-af2f-72feabc4c7bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:31.133396 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.133372    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-home" (OuterVolumeSpecName: "home") pod "7a2f28d5-b884-468d-af2f-72feabc4c7bf" (UID: "7a2f28d5-b884-468d-af2f-72feabc4c7bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:31.133712 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.133660    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a2f28d5-b884-468d-af2f-72feabc4c7bf" (UID: "7a2f28d5-b884-468d-af2f-72feabc4c7bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:31.133803 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.133747    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:05:31.133803 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.133766    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:05:31.135131 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.135108    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2f28d5-b884-468d-af2f-72feabc4c7bf-kube-api-access-69pg2" (OuterVolumeSpecName: "kube-api-access-69pg2") pod "7a2f28d5-b884-468d-af2f-72feabc4c7bf" (UID: "7a2f28d5-b884-468d-af2f-72feabc4c7bf"). InnerVolumeSpecName "kube-api-access-69pg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:05:31.135260 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.135155    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a2f28d5-b884-468d-af2f-72feabc4c7bf" (UID: "7a2f28d5-b884-468d-af2f-72feabc4c7bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:05:31.135260 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.135180    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-dshm" (OuterVolumeSpecName: "dshm") pod "7a2f28d5-b884-468d-af2f-72feabc4c7bf" (UID: "7a2f28d5-b884-468d-af2f-72feabc4c7bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:31.234379 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.234350    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69pg2\" (UniqueName: \"kubernetes.io/projected/7a2f28d5-b884-468d-af2f-72feabc4c7bf-kube-api-access-69pg2\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:05:31.234379 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.234372    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:05:31.234379 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.234380    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2f28d5-b884-468d-af2f-72feabc4c7bf-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:05:31.234559 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.234391    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2f28d5-b884-468d-af2f-72feabc4c7bf-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:05:31.432929 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.432844    2550 generic.go:356] "Generic (PLEG): container finished" podID="7a2f28d5-b884-468d-af2f-72feabc4c7bf" containerID="9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063" exitCode=0
Jul 06 21:05:31.432929 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.432886    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv" event={"ID":"7a2f28d5-b884-468d-af2f-72feabc4c7bf","Type":"ContainerDied","Data":"9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063"}
Jul 06 21:05:31.432929 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.432892    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"
Jul 06 21:05:31.433130 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.432939    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv" event={"ID":"7a2f28d5-b884-468d-af2f-72feabc4c7bf","Type":"ContainerDied","Data":"9da1fbf8aab2804f98180fceb3a5d281651ca7532e61280a8cee49236d098ce0"}
Jul 06 21:05:31.433130 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.432970    2550 scope.go:117] "RemoveContainer" containerID="9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063"
Jul 06 21:05:31.441257 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.441240    2550 scope.go:117] "RemoveContainer" containerID="9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063"
Jul 06 21:05:31.441569 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:05:31.441550    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063\": container with ID starting with 9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063 not found: ID does not exist" containerID="9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063"
Jul 06 21:05:31.441631 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.441587    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063"} err="failed to get container status \"9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063\": rpc error: code = NotFound desc = could not find container \"9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063\": container with ID starting with 9399368a6f5a2c0436dbe853fe880241cbf7cd35eebee67e24f794990b112063 not found: ID does not exist"
Jul 06 21:05:31.454662 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.454641    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"]
Jul 06 21:05:31.457022 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.457003    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv"]
Jul 06 21:05:31.473273 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:31.473251    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a2f28d5-b884-468d-af2f-72feabc4c7bf" path="/var/lib/kubelet/pods/7a2f28d5-b884-468d-af2f-72feabc4c7bf/volumes"
Jul 06 21:05:40.834340 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.834303    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"]
Jul 06 21:05:40.834718 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.834627    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2f28d5-b884-468d-af2f-72feabc4c7bf" containerName="main"
Jul 06 21:05:40.834718 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.834637    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2f28d5-b884-468d-af2f-72feabc4c7bf" containerName="main"
Jul 06 21:05:40.834718 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.834694    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2f28d5-b884-468d-af2f-72feabc4c7bf" containerName="main"
Jul 06 21:05:40.838694 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.838674    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:40.841303 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.841284    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 06 21:05:40.841423 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.841336    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:05:40.842373 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.842357    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-5p6cd\""
Jul 06 21:05:40.851022 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.851000    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"]
Jul 06 21:05:40.910107 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.910081    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:40.910234 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.910122    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:40.910234 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.910201    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:40.910311 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:40.910241    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdgxz\" (UniqueName: \"kubernetes.io/projected/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kube-api-access-zdgxz\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:41.011375 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.011339    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:41.011511 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.011430    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:41.011511 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.011475    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdgxz\" (UniqueName: \"kubernetes.io/projected/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kube-api-access-zdgxz\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:41.011612 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.011516    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:41.011903 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.011885    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:41.011947 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.011900    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:41.013870 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.013854    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:41.019871 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.019847    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdgxz\" (UniqueName: \"kubernetes.io/projected/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kube-api-access-zdgxz\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:41.148165 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.148074    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:41.290553 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.290377    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"]
Jul 06 21:05:41.294110 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:05:41.294084    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd632cdbb_1f76_45ea_b5b2_72ca7e984336.slice/crio-5d7840d672f4de44f08ff926ad0ceccb6166f3f4d2db8c7995c74c150ce79af2 WatchSource:0}: Error finding container 5d7840d672f4de44f08ff926ad0ceccb6166f3f4d2db8c7995c74c150ce79af2: Status 404 returned error can't find the container with id 5d7840d672f4de44f08ff926ad0ceccb6166f3f4d2db8c7995c74c150ce79af2
Jul 06 21:05:41.478133 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.478047    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68" event={"ID":"d632cdbb-1f76-45ea-b5b2-72ca7e984336","Type":"ContainerStarted","Data":"909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74"}
Jul 06 21:05:41.478133 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:41.478098    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68" event={"ID":"d632cdbb-1f76-45ea-b5b2-72ca7e984336","Type":"ContainerStarted","Data":"5d7840d672f4de44f08ff926ad0ceccb6166f3f4d2db8c7995c74c150ce79af2"}
Jul 06 21:05:42.484000 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:42.483966    2550 generic.go:356] "Generic (PLEG): container finished" podID="d632cdbb-1f76-45ea-b5b2-72ca7e984336" containerID="909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74" exitCode=0
Jul 06 21:05:42.484408 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:42.484048    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68" event={"ID":"d632cdbb-1f76-45ea-b5b2-72ca7e984336","Type":"ContainerDied","Data":"909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74"}
Jul 06 21:05:43.489751 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:43.489713    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68" event={"ID":"d632cdbb-1f76-45ea-b5b2-72ca7e984336","Type":"ContainerStarted","Data":"c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc"}
Jul 06 21:05:43.490229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:43.489869    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:05:43.511287 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:05:43.511238    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68" podStartSLOduration=2.443518086 podStartE2EDuration="3.511223134s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.067705048s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:05:43.510356134 +0000 UTC m=+1856.451659983" watchObservedRunningTime="2026-07-06 21:05:43.511223134 +0000 UTC m=+1856.452526980"
Jul 06 21:06:14.497577 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:06:14.497500    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:08:06.480918 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.480889    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"]
Jul 06 21:08:06.489538 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:08:06.489507    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:08:06.489538 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:08:06.489538 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:08:06.489538 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b" cmd=["/bin/sleep","15"]
Jul 06 21:08:06.489823 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.489551    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt" message="" err=<
Jul 06 21:08:06.489823 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:08:06.489823 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:08:06.489823 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:08:06.489823 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:08:06.489564    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:08:06.489823 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:08:06.489823 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:08:06.489823 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt" podUID="97883cf5-0738-4698-b483-05a36edf622c" containerName="main" containerID="cri-o://c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b"
Jul 06 21:08:06.489823 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.489607    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt" podUID="97883cf5-0738-4698-b483-05a36edf622c" containerName="main" containerID="cri-o://c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b" gracePeriod=60
Jul 06 21:08:06.743119 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.743059    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:08:06.848050 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.848021    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97883cf5-0738-4698-b483-05a36edf622c-tls-certs\") pod \"97883cf5-0738-4698-b483-05a36edf622c\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") "
Jul 06 21:08:06.848050 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.848055    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m484f\" (UniqueName: \"kubernetes.io/projected/97883cf5-0738-4698-b483-05a36edf622c-kube-api-access-m484f\") pod \"97883cf5-0738-4698-b483-05a36edf622c\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") "
Jul 06 21:08:06.848267 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.848097    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/97883cf5-0738-4698-b483-05a36edf622c-tokenizer-uds\") pod \"97883cf5-0738-4698-b483-05a36edf622c\" (UID: \"97883cf5-0738-4698-b483-05a36edf622c\") "
Jul 06 21:08:06.848402 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.848380    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97883cf5-0738-4698-b483-05a36edf622c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "97883cf5-0738-4698-b483-05a36edf622c" (UID: "97883cf5-0738-4698-b483-05a36edf622c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:06.850253 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.850223    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97883cf5-0738-4698-b483-05a36edf622c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97883cf5-0738-4698-b483-05a36edf622c" (UID: "97883cf5-0738-4698-b483-05a36edf622c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:08:06.850253 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.850244    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97883cf5-0738-4698-b483-05a36edf622c-kube-api-access-m484f" (OuterVolumeSpecName: "kube-api-access-m484f") pod "97883cf5-0738-4698-b483-05a36edf622c" (UID: "97883cf5-0738-4698-b483-05a36edf622c"). InnerVolumeSpecName "kube-api-access-m484f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:08:06.948856 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.948787    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/97883cf5-0738-4698-b483-05a36edf622c-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:08:06.948856 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.948840    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97883cf5-0738-4698-b483-05a36edf622c-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:08:06.948856 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:06.948850    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m484f\" (UniqueName: \"kubernetes.io/projected/97883cf5-0738-4698-b483-05a36edf622c-kube-api-access-m484f\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:08:07.112085 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:07.112050    2550 generic.go:356] "Generic (PLEG): container finished" podID="97883cf5-0738-4698-b483-05a36edf622c" containerID="c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b" exitCode=0
Jul 06 21:08:07.112284 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:07.112100    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"
Jul 06 21:08:07.112284 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:07.112133    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt" event={"ID":"97883cf5-0738-4698-b483-05a36edf622c","Type":"ContainerDied","Data":"c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b"}
Jul 06 21:08:07.112284 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:07.112183    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt" event={"ID":"97883cf5-0738-4698-b483-05a36edf622c","Type":"ContainerDied","Data":"eac64ccc9c911351522beff50c986c2de2bd202343b177bc043465f0575ee8aa"}
Jul 06 21:08:07.112284 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:07.112203    2550 scope.go:117] "RemoveContainer" containerID="c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b"
Jul 06 21:08:07.120949 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:07.120935    2550 scope.go:117] "RemoveContainer" containerID="c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b"
Jul 06 21:08:07.121190 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:08:07.121171    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b\": container with ID starting with c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b not found: ID does not exist" containerID="c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b"
Jul 06 21:08:07.121232 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:07.121197    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b"} err="failed to get container status \"c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b\": rpc error: code = NotFound desc = could not find container \"c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b\": container with ID starting with c394850d46f53ce01b15378f49acd4f37d788f4db2995ac0561500ea7e5aa85b not found: ID does not exist"
Jul 06 21:08:07.135001 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:07.134977    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"]
Jul 06 21:08:07.139592 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:07.139573    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt"]
Jul 06 21:08:07.473852 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:07.473766    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97883cf5-0738-4698-b483-05a36edf622c" path="/var/lib/kubelet/pods/97883cf5-0738-4698-b483-05a36edf622c/volumes"
Jul 06 21:08:26.233081 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.233045    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"]
Jul 06 21:08:26.241593 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:08:26.241551    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:08:26.241593 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:08:26.241593 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:08:26.241593 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc" cmd=["/bin/sleep","15"]
Jul 06 21:08:26.241863 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.241617    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68" message="" err=<
Jul 06 21:08:26.241863 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:08:26.241863 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:08:26.241863 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:08:26.241863 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:08:26.241648    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:08:26.241863 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:08:26.241863 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:08:26.241863 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68" podUID="d632cdbb-1f76-45ea-b5b2-72ca7e984336" containerName="main" containerID="cri-o://c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc"
Jul 06 21:08:26.241863 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.241709    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68" podUID="d632cdbb-1f76-45ea-b5b2-72ca7e984336" containerName="main" containerID="cri-o://c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc" gracePeriod=60
Jul 06 21:08:26.492977 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.492913    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:08:26.610703 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.610672    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tokenizer-uds\") pod \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") "
Jul 06 21:08:26.610905 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.610761    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tls-certs\") pod \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") "
Jul 06 21:08:26.610905 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.610789    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kserve-provision-location\") pod \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") "
Jul 06 21:08:26.610905 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.610853    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdgxz\" (UniqueName: \"kubernetes.io/projected/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kube-api-access-zdgxz\") pod \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\" (UID: \"d632cdbb-1f76-45ea-b5b2-72ca7e984336\") "
Jul 06 21:08:26.611065 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.610979    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d632cdbb-1f76-45ea-b5b2-72ca7e984336" (UID: "d632cdbb-1f76-45ea-b5b2-72ca7e984336"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:26.611130 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.611112    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:08:26.611475 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.611451    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d632cdbb-1f76-45ea-b5b2-72ca7e984336" (UID: "d632cdbb-1f76-45ea-b5b2-72ca7e984336"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:26.612851 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.612828    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d632cdbb-1f76-45ea-b5b2-72ca7e984336" (UID: "d632cdbb-1f76-45ea-b5b2-72ca7e984336"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:08:26.612931 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.612866    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kube-api-access-zdgxz" (OuterVolumeSpecName: "kube-api-access-zdgxz") pod "d632cdbb-1f76-45ea-b5b2-72ca7e984336" (UID: "d632cdbb-1f76-45ea-b5b2-72ca7e984336"). InnerVolumeSpecName "kube-api-access-zdgxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:08:26.712514 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.712485    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d632cdbb-1f76-45ea-b5b2-72ca7e984336-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:08:26.712630 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.712520    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:08:26.712630 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:26.712533    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdgxz\" (UniqueName: \"kubernetes.io/projected/d632cdbb-1f76-45ea-b5b2-72ca7e984336-kube-api-access-zdgxz\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:08:27.207675 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.207638    2550 generic.go:356] "Generic (PLEG): container finished" podID="d632cdbb-1f76-45ea-b5b2-72ca7e984336" containerID="c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc" exitCode=0
Jul 06 21:08:27.207861 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.207708    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"
Jul 06 21:08:27.207861 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.207706    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68" event={"ID":"d632cdbb-1f76-45ea-b5b2-72ca7e984336","Type":"ContainerDied","Data":"c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc"}
Jul 06 21:08:27.207861 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.207787    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68" event={"ID":"d632cdbb-1f76-45ea-b5b2-72ca7e984336","Type":"ContainerDied","Data":"5d7840d672f4de44f08ff926ad0ceccb6166f3f4d2db8c7995c74c150ce79af2"}
Jul 06 21:08:27.207861 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.207823    2550 scope.go:117] "RemoveContainer" containerID="c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc"
Jul 06 21:08:27.216998 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.216980    2550 scope.go:117] "RemoveContainer" containerID="909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74"
Jul 06 21:08:27.224562 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.224548    2550 scope.go:117] "RemoveContainer" containerID="c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc"
Jul 06 21:08:27.224794 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:08:27.224775    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc\": container with ID starting with c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc not found: ID does not exist" containerID="c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc"
Jul 06 21:08:27.224873 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.224801    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc"} err="failed to get container status \"c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc\": rpc error: code = NotFound desc = could not find container \"c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc\": container with ID starting with c1831fe6083466ade686435ac1621179676b13a7b0d08e494eff9bfc4f18c4cc not found: ID does not exist"
Jul 06 21:08:27.224924 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.224876    2550 scope.go:117] "RemoveContainer" containerID="909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74"
Jul 06 21:08:27.225094 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:08:27.225079    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74\": container with ID starting with 909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74 not found: ID does not exist" containerID="909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74"
Jul 06 21:08:27.225148 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.225098    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74"} err="failed to get container status \"909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74\": rpc error: code = NotFound desc = could not find container \"909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74\": container with ID starting with 909db45d95a7e023fc07c8ae6e362f46581f2e3e5c83831739fcf625a24d8b74 not found: ID does not exist"
Jul 06 21:08:27.230587 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.230566    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"]
Jul 06 21:08:27.232985 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.232966    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68"]
Jul 06 21:08:27.473819 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:27.473728    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d632cdbb-1f76-45ea-b5b2-72ca7e984336" path="/var/lib/kubelet/pods/d632cdbb-1f76-45ea-b5b2-72ca7e984336/volumes"
Jul 06 21:08:35.530201 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.530171    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"]
Jul 06 21:08:35.530650 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.530472    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d632cdbb-1f76-45ea-b5b2-72ca7e984336" containerName="main"
Jul 06 21:08:35.530650 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.530483    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d632cdbb-1f76-45ea-b5b2-72ca7e984336" containerName="main"
Jul 06 21:08:35.530650 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.530494    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d632cdbb-1f76-45ea-b5b2-72ca7e984336" containerName="storage-initializer"
Jul 06 21:08:35.530650 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.530500    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d632cdbb-1f76-45ea-b5b2-72ca7e984336" containerName="storage-initializer"
Jul 06 21:08:35.530650 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.530520    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97883cf5-0738-4698-b483-05a36edf622c" containerName="main"
Jul 06 21:08:35.530650 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.530525    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="97883cf5-0738-4698-b483-05a36edf622c" containerName="main"
Jul 06 21:08:35.530650 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.530573    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="97883cf5-0738-4698-b483-05a36edf622c" containerName="main"
Jul 06 21:08:35.530650 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.530579    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d632cdbb-1f76-45ea-b5b2-72ca7e984336" containerName="main"
Jul 06 21:08:35.534586 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.534568    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.539037 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.539015    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-j79p9\""
Jul 06 21:08:35.539172 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.539047    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 06 21:08:35.539172 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.539135    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:08:35.539378 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.539361    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:08:35.539446 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.539382    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:08:35.548896 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.548874    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"]
Jul 06 21:08:35.690160 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.690136    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pd6t\" (UniqueName: \"kubernetes.io/projected/0920e84a-7f35-47de-aa6d-c7905023291e-kube-api-access-4pd6t\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.690309 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.690185    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0920e84a-7f35-47de-aa6d-c7905023291e-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.690309 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.690228    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0920e84a-7f35-47de-aa6d-c7905023291e-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.791100 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.791032    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0920e84a-7f35-47de-aa6d-c7905023291e-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.791100 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.791076    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pd6t\" (UniqueName: \"kubernetes.io/projected/0920e84a-7f35-47de-aa6d-c7905023291e-kube-api-access-4pd6t\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.791246 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.791135    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0920e84a-7f35-47de-aa6d-c7905023291e-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.791569 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.791536    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0920e84a-7f35-47de-aa6d-c7905023291e-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.793589 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.793572    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0920e84a-7f35-47de-aa6d-c7905023291e-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.803626 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.803605    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pd6t\" (UniqueName: \"kubernetes.io/projected/0920e84a-7f35-47de-aa6d-c7905023291e-kube-api-access-4pd6t\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.843803 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.843772    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:35.976290 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:35.976258    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"]
Jul 06 21:08:35.978561 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:08:35.978530    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0920e84a_7f35_47de_aa6d_c7905023291e.slice/crio-f8f4395d066c6cff596428ad9867ae2f8e31b0d14e8fd4219f7744a14ad44f96 WatchSource:0}: Error finding container f8f4395d066c6cff596428ad9867ae2f8e31b0d14e8fd4219f7744a14ad44f96: Status 404 returned error can't find the container with id f8f4395d066c6cff596428ad9867ae2f8e31b0d14e8fd4219f7744a14ad44f96
Jul 06 21:08:36.249793 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:36.249753    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5" event={"ID":"0920e84a-7f35-47de-aa6d-c7905023291e","Type":"ContainerStarted","Data":"ce9043839aa54a8e8f48ece0aee46a780d7e97ec93f1c0934561d93371252eec"}
Jul 06 21:08:36.249989 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:36.249849    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:08:36.249989 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:36.249873    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5" event={"ID":"0920e84a-7f35-47de-aa6d-c7905023291e","Type":"ContainerStarted","Data":"f8f4395d066c6cff596428ad9867ae2f8e31b0d14e8fd4219f7744a14ad44f96"}
Jul 06 21:08:36.275074 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:36.275028    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5" podStartSLOduration=1.275012973 podStartE2EDuration="1.275012973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:36.272926617 +0000 UTC m=+2029.214230486" watchObservedRunningTime="2026-07-06 21:08:36.275012973 +0000 UTC m=+2029.216316860"
Jul 06 21:08:53.702616 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.702580    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"]
Jul 06 21:08:53.706082 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.706060    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.709695 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.709674    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 06 21:08:53.709834 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.709674    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:08:53.709834 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.709675    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-sz52l\""
Jul 06 21:08:53.721097 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.721074    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"]
Jul 06 21:08:53.742406 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.742377    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.742553 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.742410    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2b5j\" (UniqueName: \"kubernetes.io/projected/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kube-api-access-j2b5j\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.742553 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.742517    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.742667 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.742575    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.843911 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.843880    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.844061 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.843929    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.844196 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.844171    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.844263 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.844216    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2b5j\" (UniqueName: \"kubernetes.io/projected/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kube-api-access-j2b5j\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.844315 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.844263    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.844545 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.844525    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.846259 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.846235    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:53.852694 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:53.852671    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2b5j\" (UniqueName: \"kubernetes.io/projected/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kube-api-access-j2b5j\") pod \"custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:54.016301 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:54.016226    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:54.151424 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:54.151399    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"]
Jul 06 21:08:54.153407 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:08:54.153382    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1f3157_3e71_401a_8d48_4ffeefaa77cb.slice/crio-0fdcec5ec0efc96d2a5b107ddd7cb8dc86d5a97f13a5abb38d0057a010e6e84a WatchSource:0}: Error finding container 0fdcec5ec0efc96d2a5b107ddd7cb8dc86d5a97f13a5abb38d0057a010e6e84a: Status 404 returned error can't find the container with id 0fdcec5ec0efc96d2a5b107ddd7cb8dc86d5a97f13a5abb38d0057a010e6e84a
Jul 06 21:08:54.325579 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:54.325547    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr" event={"ID":"2d1f3157-3e71-401a-8d48-4ffeefaa77cb","Type":"ContainerStarted","Data":"3e8cd29b9d98325eaf1b27e7da576cf50eb25e903906867b66d239fc07765ab7"}
Jul 06 21:08:54.325790 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:54.325601    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr" event={"ID":"2d1f3157-3e71-401a-8d48-4ffeefaa77cb","Type":"ContainerStarted","Data":"0fdcec5ec0efc96d2a5b107ddd7cb8dc86d5a97f13a5abb38d0057a010e6e84a"}
Jul 06 21:08:55.331700 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:55.331667    2550 generic.go:356] "Generic (PLEG): container finished" podID="2d1f3157-3e71-401a-8d48-4ffeefaa77cb" containerID="3e8cd29b9d98325eaf1b27e7da576cf50eb25e903906867b66d239fc07765ab7" exitCode=0
Jul 06 21:08:55.332127 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:55.331721    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr" event={"ID":"2d1f3157-3e71-401a-8d48-4ffeefaa77cb","Type":"ContainerDied","Data":"3e8cd29b9d98325eaf1b27e7da576cf50eb25e903906867b66d239fc07765ab7"}
Jul 06 21:08:56.338256 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:56.338216    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr" event={"ID":"2d1f3157-3e71-401a-8d48-4ffeefaa77cb","Type":"ContainerStarted","Data":"702cfdae2f734e83d6db45a27ed0da43a8c5193071e4d6b26bc4e8d701504039"}
Jul 06 21:08:56.338654 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:56.338358    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:08:56.370846 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:08:56.370777    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr" podStartSLOduration=2.714399714 podStartE2EDuration="3.370762677s" totalImagesPullingTime="0s" totalInitContainerRuntime="656.362963ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:56.368613997 +0000 UTC m=+2049.309917848" watchObservedRunningTime="2026-07-06 21:08:56.370762677 +0000 UTC m=+2049.312066523"
Jul 06 21:09:07.257473 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:09:07.257442    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:09:27.344441 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:09:27.344367    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:11:48.942364 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:48.942329    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"]
Jul 06 21:11:48.951060 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:11:48.951022    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:11:48.951060 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:11:48.951060 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:11:48.951060 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="702cfdae2f734e83d6db45a27ed0da43a8c5193071e4d6b26bc4e8d701504039" cmd=["/bin/sleep","15"]
Jul 06 21:11:48.951376 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:48.951078    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr" message="" err=<
Jul 06 21:11:48.951376 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:11:48.951376 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:11:48.951376 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:11:48.951376 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:11:48.951094    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:11:48.951376 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:11:48.951376 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:11:48.951376 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr" podUID="2d1f3157-3e71-401a-8d48-4ffeefaa77cb" containerName="main" containerID="cri-o://702cfdae2f734e83d6db45a27ed0da43a8c5193071e4d6b26bc4e8d701504039"
Jul 06 21:11:48.951376 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:48.951137    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr" podUID="2d1f3157-3e71-401a-8d48-4ffeefaa77cb" containerName="main" containerID="cri-o://702cfdae2f734e83d6db45a27ed0da43a8c5193071e4d6b26bc4e8d701504039" gracePeriod=60
Jul 06 21:11:49.091907 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.091871    2550 generic.go:356] "Generic (PLEG): container finished" podID="2d1f3157-3e71-401a-8d48-4ffeefaa77cb" containerID="702cfdae2f734e83d6db45a27ed0da43a8c5193071e4d6b26bc4e8d701504039" exitCode=0
Jul 06 21:11:49.092061 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.091941    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr" event={"ID":"2d1f3157-3e71-401a-8d48-4ffeefaa77cb","Type":"ContainerDied","Data":"702cfdae2f734e83d6db45a27ed0da43a8c5193071e4d6b26bc4e8d701504039"}
Jul 06 21:11:49.204246 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.204190    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:11:49.280676 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.280645    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tls-certs\") pod \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") "
Jul 06 21:11:49.280878 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.280703    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2b5j\" (UniqueName: \"kubernetes.io/projected/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kube-api-access-j2b5j\") pod \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") "
Jul 06 21:11:49.280878 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.280733    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tokenizer-uds\") pod \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") "
Jul 06 21:11:49.280878 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.280750    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kserve-provision-location\") pod \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\" (UID: \"2d1f3157-3e71-401a-8d48-4ffeefaa77cb\") "
Jul 06 21:11:49.281094 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.281070    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2d1f3157-3e71-401a-8d48-4ffeefaa77cb" (UID: "2d1f3157-3e71-401a-8d48-4ffeefaa77cb"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:49.281493 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.281471    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d1f3157-3e71-401a-8d48-4ffeefaa77cb" (UID: "2d1f3157-3e71-401a-8d48-4ffeefaa77cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:49.282750 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.282732    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kube-api-access-j2b5j" (OuterVolumeSpecName: "kube-api-access-j2b5j") pod "2d1f3157-3e71-401a-8d48-4ffeefaa77cb" (UID: "2d1f3157-3e71-401a-8d48-4ffeefaa77cb"). InnerVolumeSpecName "kube-api-access-j2b5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:11:49.282861 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.282844    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d1f3157-3e71-401a-8d48-4ffeefaa77cb" (UID: "2d1f3157-3e71-401a-8d48-4ffeefaa77cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:11:49.382012 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.381979    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:11:49.382012 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.382006    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2b5j\" (UniqueName: \"kubernetes.io/projected/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kube-api-access-j2b5j\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:11:49.382012 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.382020    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:11:49.382241 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:49.382033    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1f3157-3e71-401a-8d48-4ffeefaa77cb-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:11:50.097465 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:50.097430    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr" event={"ID":"2d1f3157-3e71-401a-8d48-4ffeefaa77cb","Type":"ContainerDied","Data":"0fdcec5ec0efc96d2a5b107ddd7cb8dc86d5a97f13a5abb38d0057a010e6e84a"}
Jul 06 21:11:50.097905 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:50.097480    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"
Jul 06 21:11:50.097905 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:50.097489    2550 scope.go:117] "RemoveContainer" containerID="702cfdae2f734e83d6db45a27ed0da43a8c5193071e4d6b26bc4e8d701504039"
Jul 06 21:11:50.106063 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:50.106046    2550 scope.go:117] "RemoveContainer" containerID="3e8cd29b9d98325eaf1b27e7da576cf50eb25e903906867b66d239fc07765ab7"
Jul 06 21:11:50.119207 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:50.119177    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"]
Jul 06 21:11:50.122989 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:50.122967    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr"]
Jul 06 21:11:51.474679 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:51.474642    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1f3157-3e71-401a-8d48-4ffeefaa77cb" path="/var/lib/kubelet/pods/2d1f3157-3e71-401a-8d48-4ffeefaa77cb/volumes"
Jul 06 21:11:54.871894 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:54.871838    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"]
Jul 06 21:11:54.880405 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:11:54.880361    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:11:54.880405 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:11:54.880405 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:11:54.880405 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="ce9043839aa54a8e8f48ece0aee46a780d7e97ec93f1c0934561d93371252eec" cmd=["/bin/sleep","15"]
Jul 06 21:11:54.880630 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:54.880418    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5" message="" err=<
Jul 06 21:11:54.880630 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:11:54.880630 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:11:54.880630 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:11:54.880630 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:11:54.880430    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:11:54.880630 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:11:54.880630 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:11:54.880630 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5" podUID="0920e84a-7f35-47de-aa6d-c7905023291e" containerName="main" containerID="cri-o://ce9043839aa54a8e8f48ece0aee46a780d7e97ec93f1c0934561d93371252eec"
Jul 06 21:11:54.880630 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:54.880472    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5" podUID="0920e84a-7f35-47de-aa6d-c7905023291e" containerName="main" containerID="cri-o://ce9043839aa54a8e8f48ece0aee46a780d7e97ec93f1c0934561d93371252eec" gracePeriod=60
Jul 06 21:11:55.122773 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.122704    2550 generic.go:356] "Generic (PLEG): container finished" podID="0920e84a-7f35-47de-aa6d-c7905023291e" containerID="ce9043839aa54a8e8f48ece0aee46a780d7e97ec93f1c0934561d93371252eec" exitCode=0
Jul 06 21:11:55.122773 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.122740    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5" event={"ID":"0920e84a-7f35-47de-aa6d-c7905023291e","Type":"ContainerDied","Data":"ce9043839aa54a8e8f48ece0aee46a780d7e97ec93f1c0934561d93371252eec"}
Jul 06 21:11:55.122959 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.122849    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5" event={"ID":"0920e84a-7f35-47de-aa6d-c7905023291e","Type":"ContainerDied","Data":"f8f4395d066c6cff596428ad9867ae2f8e31b0d14e8fd4219f7744a14ad44f96"}
Jul 06 21:11:55.122959 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.122871    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8f4395d066c6cff596428ad9867ae2f8e31b0d14e8fd4219f7744a14ad44f96"
Jul 06 21:11:55.125066 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.125049    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:11:55.226239 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.226215    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0920e84a-7f35-47de-aa6d-c7905023291e-tokenizer-uds\") pod \"0920e84a-7f35-47de-aa6d-c7905023291e\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") "
Jul 06 21:11:55.226373 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.226261    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0920e84a-7f35-47de-aa6d-c7905023291e-tls-certs\") pod \"0920e84a-7f35-47de-aa6d-c7905023291e\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") "
Jul 06 21:11:55.226373 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.226290    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pd6t\" (UniqueName: \"kubernetes.io/projected/0920e84a-7f35-47de-aa6d-c7905023291e-kube-api-access-4pd6t\") pod \"0920e84a-7f35-47de-aa6d-c7905023291e\" (UID: \"0920e84a-7f35-47de-aa6d-c7905023291e\") "
Jul 06 21:11:55.226455 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.226442    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0920e84a-7f35-47de-aa6d-c7905023291e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0920e84a-7f35-47de-aa6d-c7905023291e" (UID: "0920e84a-7f35-47de-aa6d-c7905023291e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:55.228214 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.228196    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0920e84a-7f35-47de-aa6d-c7905023291e-kube-api-access-4pd6t" (OuterVolumeSpecName: "kube-api-access-4pd6t") pod "0920e84a-7f35-47de-aa6d-c7905023291e" (UID: "0920e84a-7f35-47de-aa6d-c7905023291e"). InnerVolumeSpecName "kube-api-access-4pd6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:11:55.228390 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.228373    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0920e84a-7f35-47de-aa6d-c7905023291e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0920e84a-7f35-47de-aa6d-c7905023291e" (UID: "0920e84a-7f35-47de-aa6d-c7905023291e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:11:55.327158 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.327132    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pd6t\" (UniqueName: \"kubernetes.io/projected/0920e84a-7f35-47de-aa6d-c7905023291e-kube-api-access-4pd6t\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:11:55.327158 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.327155    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0920e84a-7f35-47de-aa6d-c7905023291e-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:11:55.327309 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:55.327165    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0920e84a-7f35-47de-aa6d-c7905023291e-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:11:56.127438 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:56.127403    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"
Jul 06 21:11:56.153486 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:56.153455    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"]
Jul 06 21:11:56.156593 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:56.156568    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5"]
Jul 06 21:11:57.473109 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:11:57.473078    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0920e84a-7f35-47de-aa6d-c7905023291e" path="/var/lib/kubelet/pods/0920e84a-7f35-47de-aa6d-c7905023291e/volumes"
Jul 06 21:12:16.953525 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.953438    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"]
Jul 06 21:12:16.953924 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.953839    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1f3157-3e71-401a-8d48-4ffeefaa77cb" containerName="main"
Jul 06 21:12:16.953924 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.953854    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1f3157-3e71-401a-8d48-4ffeefaa77cb" containerName="main"
Jul 06 21:12:16.953924 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.953868    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0920e84a-7f35-47de-aa6d-c7905023291e" containerName="main"
Jul 06 21:12:16.953924 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.953874    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0920e84a-7f35-47de-aa6d-c7905023291e" containerName="main"
Jul 06 21:12:16.953924 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.953880    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1f3157-3e71-401a-8d48-4ffeefaa77cb" containerName="storage-initializer"
Jul 06 21:12:16.953924 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.953886    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1f3157-3e71-401a-8d48-4ffeefaa77cb" containerName="storage-initializer"
Jul 06 21:12:16.954111 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.953948    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0920e84a-7f35-47de-aa6d-c7905023291e" containerName="main"
Jul 06 21:12:16.954111 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.953959    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1f3157-3e71-401a-8d48-4ffeefaa77cb" containerName="main"
Jul 06 21:12:16.957404 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.957383    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:16.960063 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.960041    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:12:16.960168 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.960105    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-vt6jn\""
Jul 06 21:12:16.961302 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.961275    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 06 21:12:16.961420 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.961343    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:12:16.961420 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.961362    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:12:16.961540 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.961489    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:12:16.969169 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:16.969099    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"]
Jul 06 21:12:17.105365 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.105332    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.105365 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.105371    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e753341d-9ed6-4709-9196-2d5d8079e543-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.105589 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.105407    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.105589 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.105445    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvmf\" (UniqueName: \"kubernetes.io/projected/e753341d-9ed6-4709-9196-2d5d8079e543-kube-api-access-wsvmf\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.206751 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.206682    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e753341d-9ed6-4709-9196-2d5d8079e543-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.206751 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.206722    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.206989 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.206752    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvmf\" (UniqueName: \"kubernetes.io/projected/e753341d-9ed6-4709-9196-2d5d8079e543-kube-api-access-wsvmf\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.206989 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.206802    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.207099 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.207082    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.207157 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.207107    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.209228 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.209204    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e753341d-9ed6-4709-9196-2d5d8079e543-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.216254 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.216235    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvmf\" (UniqueName: \"kubernetes.io/projected/e753341d-9ed6-4709-9196-2d5d8079e543-kube-api-access-wsvmf\") pod \"router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.267585 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.267563    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:17.421242 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:12:17.421212    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode753341d_9ed6_4709_9196_2d5d8079e543.slice/crio-23506c1d18fe3aa73c0dec3d9820398aef2870a03943930d50b9af2efe2efb5f WatchSource:0}: Error finding container 23506c1d18fe3aa73c0dec3d9820398aef2870a03943930d50b9af2efe2efb5f: Status 404 returned error can't find the container with id 23506c1d18fe3aa73c0dec3d9820398aef2870a03943930d50b9af2efe2efb5f
Jul 06 21:12:17.421376 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:17.421288    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"]
Jul 06 21:12:18.227412 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:18.227376    2550 generic.go:356] "Generic (PLEG): container finished" podID="e753341d-9ed6-4709-9196-2d5d8079e543" containerID="9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8" exitCode=0
Jul 06 21:12:18.227733 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:18.227458    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g" event={"ID":"e753341d-9ed6-4709-9196-2d5d8079e543","Type":"ContainerDied","Data":"9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8"}
Jul 06 21:12:18.227733 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:18.227511    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g" event={"ID":"e753341d-9ed6-4709-9196-2d5d8079e543","Type":"ContainerStarted","Data":"23506c1d18fe3aa73c0dec3d9820398aef2870a03943930d50b9af2efe2efb5f"}
Jul 06 21:12:19.233415 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:19.233380    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g" event={"ID":"e753341d-9ed6-4709-9196-2d5d8079e543","Type":"ContainerStarted","Data":"91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4"}
Jul 06 21:12:19.233796 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:19.233452    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:12:19.259642 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:19.259594    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g" podStartSLOduration=2.645659242 podStartE2EDuration="3.259578605s" totalImagesPullingTime="0s" totalInitContainerRuntime="613.919363ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:12:19.2571754 +0000 UTC m=+2252.198479237" watchObservedRunningTime="2026-07-06 21:12:19.259578605 +0000 UTC m=+2252.200882464"
Jul 06 21:12:38.947330 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:38.947295    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"]
Jul 06 21:12:38.951034 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:38.951014    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:38.955313 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:38.955291    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 06 21:12:38.955417 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:38.955291    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-dl7c4\""
Jul 06 21:12:38.972906 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:38.972882    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"]
Jul 06 21:12:39.090823 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.090784    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a144dbb-03c8-4613-a1d0-3e489726e141-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.090969 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.090854    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xdd\" (UniqueName: \"kubernetes.io/projected/2a144dbb-03c8-4613-a1d0-3e489726e141-kube-api-access-k2xdd\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.090969 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.090887    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.090969 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.090908    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.191568 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.191539    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xdd\" (UniqueName: \"kubernetes.io/projected/2a144dbb-03c8-4613-a1d0-3e489726e141-kube-api-access-k2xdd\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.191741 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.191600    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.191741 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.191632    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.191741 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.191685    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a144dbb-03c8-4613-a1d0-3e489726e141-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.192045 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.192024    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.192119 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.192046    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.194090 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.194070    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a144dbb-03c8-4613-a1d0-3e489726e141-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.200665 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.200598    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xdd\" (UniqueName: \"kubernetes.io/projected/2a144dbb-03c8-4613-a1d0-3e489726e141-kube-api-access-k2xdd\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.260039 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.260013    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:39.401704 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:39.401670    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"]
Jul 06 21:12:39.403839 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:12:39.403783    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a144dbb_03c8_4613_a1d0_3e489726e141.slice/crio-742abe37e04d730dc59ed20c2c367307bbb769fea4f12ef0e15fb3a07fbfac30 WatchSource:0}: Error finding container 742abe37e04d730dc59ed20c2c367307bbb769fea4f12ef0e15fb3a07fbfac30: Status 404 returned error can't find the container with id 742abe37e04d730dc59ed20c2c367307bbb769fea4f12ef0e15fb3a07fbfac30
Jul 06 21:12:40.327686 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:40.327644    2550 generic.go:356] "Generic (PLEG): container finished" podID="2a144dbb-03c8-4613-a1d0-3e489726e141" containerID="c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9" exitCode=0
Jul 06 21:12:40.328136 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:40.327705    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml" event={"ID":"2a144dbb-03c8-4613-a1d0-3e489726e141","Type":"ContainerDied","Data":"c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9"}
Jul 06 21:12:40.328136 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:40.327767    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml" event={"ID":"2a144dbb-03c8-4613-a1d0-3e489726e141","Type":"ContainerStarted","Data":"742abe37e04d730dc59ed20c2c367307bbb769fea4f12ef0e15fb3a07fbfac30"}
Jul 06 21:12:41.333373 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:41.333336    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml" event={"ID":"2a144dbb-03c8-4613-a1d0-3e489726e141","Type":"ContainerStarted","Data":"9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc"}
Jul 06 21:12:41.333755 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:41.333472    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:12:41.359186 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:41.359132    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml" podStartSLOduration=2.729274985 podStartE2EDuration="3.359114821s" totalImagesPullingTime="0s" totalInitContainerRuntime="629.839836ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:12:41.358548508 +0000 UTC m=+2274.299852392" watchObservedRunningTime="2026-07-06 21:12:41.359114821 +0000 UTC m=+2274.300418669"
Jul 06 21:12:50.239494 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:12:50.239464    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:13:12.342139 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:13:12.342112    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:14:47.702377 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:47.702350    2550 scope.go:117] "RemoveContainer" containerID="ce9043839aa54a8e8f48ece0aee46a780d7e97ec93f1c0934561d93371252eec"
Jul 06 21:14:54.575971 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.575937    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"]
Jul 06 21:14:54.586264 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:14:54.586231    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:14:54.586264 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:14:54.586264 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:14:54.586264 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4" cmd=["/bin/sleep","15"]
Jul 06 21:14:54.586572 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.586278    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g" message="" err=<
Jul 06 21:14:54.586572 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:14:54.586572 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:14:54.586572 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:14:54.586572 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:14:54.586289    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:14:54.586572 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:14:54.586572 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:14:54.586572 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g" podUID="e753341d-9ed6-4709-9196-2d5d8079e543" containerName="main" containerID="cri-o://91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4"
Jul 06 21:14:54.586572 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.586325    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g" podUID="e753341d-9ed6-4709-9196-2d5d8079e543" containerName="main" containerID="cri-o://91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4" gracePeriod=60
Jul 06 21:14:54.826978 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.826921    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:14:54.916921 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.916885    2550 generic.go:356] "Generic (PLEG): container finished" podID="e753341d-9ed6-4709-9196-2d5d8079e543" containerID="91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4" exitCode=0
Jul 06 21:14:54.917096 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.916934    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"
Jul 06 21:14:54.917096 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.916971    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g" event={"ID":"e753341d-9ed6-4709-9196-2d5d8079e543","Type":"ContainerDied","Data":"91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4"}
Jul 06 21:14:54.917096 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.917013    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g" event={"ID":"e753341d-9ed6-4709-9196-2d5d8079e543","Type":"ContainerDied","Data":"23506c1d18fe3aa73c0dec3d9820398aef2870a03943930d50b9af2efe2efb5f"}
Jul 06 21:14:54.917096 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.917033    2550 scope.go:117] "RemoveContainer" containerID="91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4"
Jul 06 21:14:54.925188 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.925170    2550 scope.go:117] "RemoveContainer" containerID="9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8"
Jul 06 21:14:54.932309 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.932280    2550 scope.go:117] "RemoveContainer" containerID="91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4"
Jul 06 21:14:54.932376 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.932281    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-tokenizer-uds\") pod \"e753341d-9ed6-4709-9196-2d5d8079e543\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") "
Jul 06 21:14:54.932446 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.932422    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-kserve-provision-location\") pod \"e753341d-9ed6-4709-9196-2d5d8079e543\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") "
Jul 06 21:14:54.932497 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.932465    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsvmf\" (UniqueName: \"kubernetes.io/projected/e753341d-9ed6-4709-9196-2d5d8079e543-kube-api-access-wsvmf\") pod \"e753341d-9ed6-4709-9196-2d5d8079e543\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") "
Jul 06 21:14:54.932554 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.932491    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e753341d-9ed6-4709-9196-2d5d8079e543" (UID: "e753341d-9ed6-4709-9196-2d5d8079e543"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:54.932554 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.932512    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e753341d-9ed6-4709-9196-2d5d8079e543-tls-certs\") pod \"e753341d-9ed6-4709-9196-2d5d8079e543\" (UID: \"e753341d-9ed6-4709-9196-2d5d8079e543\") "
Jul 06 21:14:54.932669 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:14:54.932571    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4\": container with ID starting with 91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4 not found: ID does not exist" containerID="91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4"
Jul 06 21:14:54.932669 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.932599    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4"} err="failed to get container status \"91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4\": rpc error: code = NotFound desc = could not find container \"91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4\": container with ID starting with 91d869f28351b639f43a9b5b094b24a1f378b00a3f46fd5d7773a3d0f18c6ed4 not found: ID does not exist"
Jul 06 21:14:54.932669 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.932617    2550 scope.go:117] "RemoveContainer" containerID="9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8"
Jul 06 21:14:54.932867 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.932745    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:14:54.932930 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:14:54.932875    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8\": container with ID starting with 9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8 not found: ID does not exist" containerID="9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8"
Jul 06 21:14:54.932930 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.932897    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8"} err="failed to get container status \"9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8\": rpc error: code = NotFound desc = could not find container \"9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8\": container with ID starting with 9a2972e6b89f6da123db7ee1d9b9697435b44a98432c29b82d4b8685349f0fc8 not found: ID does not exist"
Jul 06 21:14:54.933225 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.933203    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e753341d-9ed6-4709-9196-2d5d8079e543" (UID: "e753341d-9ed6-4709-9196-2d5d8079e543"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:54.934442 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.934423    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e753341d-9ed6-4709-9196-2d5d8079e543-kube-api-access-wsvmf" (OuterVolumeSpecName: "kube-api-access-wsvmf") pod "e753341d-9ed6-4709-9196-2d5d8079e543" (UID: "e753341d-9ed6-4709-9196-2d5d8079e543"). InnerVolumeSpecName "kube-api-access-wsvmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:14:54.934545 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:54.934525    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e753341d-9ed6-4709-9196-2d5d8079e543-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e753341d-9ed6-4709-9196-2d5d8079e543" (UID: "e753341d-9ed6-4709-9196-2d5d8079e543"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:14:55.033536 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:55.033504    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e753341d-9ed6-4709-9196-2d5d8079e543-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:14:55.033536 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:55.033530    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsvmf\" (UniqueName: \"kubernetes.io/projected/e753341d-9ed6-4709-9196-2d5d8079e543-kube-api-access-wsvmf\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:14:55.033536 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:55.033541    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e753341d-9ed6-4709-9196-2d5d8079e543-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:14:55.260287 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:55.260264    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"]
Jul 06 21:14:55.265392 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:55.265369    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g"]
Jul 06 21:14:55.473569 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:14:55.473536    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e753341d-9ed6-4709-9196-2d5d8079e543" path="/var/lib/kubelet/pods/e753341d-9ed6-4709-9196-2d5d8079e543/volumes"
Jul 06 21:15:00.634682 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.634646    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"]
Jul 06 21:15:00.643236 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:15:00.643206    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:15:00.643236 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:15:00.643236 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:15:00.643236 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc" cmd=["/bin/sleep","15"]
Jul 06 21:15:00.643511 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.643253    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml" message="" err=<
Jul 06 21:15:00.643511 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:15:00.643511 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:15:00.643511 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:15:00.643511 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:15:00.643268    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:15:00.643511 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:15:00.643511 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:15:00.643511 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml" podUID="2a144dbb-03c8-4613-a1d0-3e489726e141" containerName="main" containerID="cri-o://9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc"
Jul 06 21:15:00.643511 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.643311    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml" podUID="2a144dbb-03c8-4613-a1d0-3e489726e141" containerName="main" containerID="cri-o://9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc" gracePeriod=60
Jul 06 21:15:00.886261 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.886207    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:15:00.944941 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.944906    2550 generic.go:356] "Generic (PLEG): container finished" podID="2a144dbb-03c8-4613-a1d0-3e489726e141" containerID="9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc" exitCode=0
Jul 06 21:15:00.945121 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.944962    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"
Jul 06 21:15:00.945121 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.944962    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml" event={"ID":"2a144dbb-03c8-4613-a1d0-3e489726e141","Type":"ContainerDied","Data":"9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc"}
Jul 06 21:15:00.945121 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.945078    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml" event={"ID":"2a144dbb-03c8-4613-a1d0-3e489726e141","Type":"ContainerDied","Data":"742abe37e04d730dc59ed20c2c367307bbb769fea4f12ef0e15fb3a07fbfac30"}
Jul 06 21:15:00.945121 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.945101    2550 scope.go:117] "RemoveContainer" containerID="9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc"
Jul 06 21:15:00.953344 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.953325    2550 scope.go:117] "RemoveContainer" containerID="c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9"
Jul 06 21:15:00.960557 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.960536    2550 scope.go:117] "RemoveContainer" containerID="9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc"
Jul 06 21:15:00.960800 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:15:00.960778    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc\": container with ID starting with 9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc not found: ID does not exist" containerID="9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc"
Jul 06 21:15:00.960917 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.960819    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc"} err="failed to get container status \"9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc\": rpc error: code = NotFound desc = could not find container \"9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc\": container with ID starting with 9642315d3d04d05d4b531e7d6a0aadb65bbef48c15c2a28dd44bcc80c510b3cc not found: ID does not exist"
Jul 06 21:15:00.960917 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.960838    2550 scope.go:117] "RemoveContainer" containerID="c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9"
Jul 06 21:15:00.961095 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:15:00.961075    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9\": container with ID starting with c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9 not found: ID does not exist" containerID="c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9"
Jul 06 21:15:00.961156 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.961106    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9"} err="failed to get container status \"c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9\": rpc error: code = NotFound desc = could not find container \"c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9\": container with ID starting with c4efca789d750abb6579ddb7b11643aa2244f3bdb49bcc93c180f11371d69ff9 not found: ID does not exist"
Jul 06 21:15:00.986475 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.986452    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a144dbb-03c8-4613-a1d0-3e489726e141-tls-certs\") pod \"2a144dbb-03c8-4613-a1d0-3e489726e141\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") "
Jul 06 21:15:00.986564 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.986497    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2xdd\" (UniqueName: \"kubernetes.io/projected/2a144dbb-03c8-4613-a1d0-3e489726e141-kube-api-access-k2xdd\") pod \"2a144dbb-03c8-4613-a1d0-3e489726e141\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") "
Jul 06 21:15:00.986564 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.986525    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-kserve-provision-location\") pod \"2a144dbb-03c8-4613-a1d0-3e489726e141\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") "
Jul 06 21:15:00.986633 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.986605    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-tokenizer-uds\") pod \"2a144dbb-03c8-4613-a1d0-3e489726e141\" (UID: \"2a144dbb-03c8-4613-a1d0-3e489726e141\") "
Jul 06 21:15:00.986835 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.986798    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2a144dbb-03c8-4613-a1d0-3e489726e141" (UID: "2a144dbb-03c8-4613-a1d0-3e489726e141"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:00.987214 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.987193    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a144dbb-03c8-4613-a1d0-3e489726e141" (UID: "2a144dbb-03c8-4613-a1d0-3e489726e141"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:00.988394 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.988371    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a144dbb-03c8-4613-a1d0-3e489726e141-kube-api-access-k2xdd" (OuterVolumeSpecName: "kube-api-access-k2xdd") pod "2a144dbb-03c8-4613-a1d0-3e489726e141" (UID: "2a144dbb-03c8-4613-a1d0-3e489726e141"). InnerVolumeSpecName "kube-api-access-k2xdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:15:00.988498 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:00.988466    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a144dbb-03c8-4613-a1d0-3e489726e141-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a144dbb-03c8-4613-a1d0-3e489726e141" (UID: "2a144dbb-03c8-4613-a1d0-3e489726e141"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:15:01.087647 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:01.087623    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:15:01.087647 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:01.087643    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a144dbb-03c8-4613-a1d0-3e489726e141-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:15:01.087789 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:01.087652    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2xdd\" (UniqueName: \"kubernetes.io/projected/2a144dbb-03c8-4613-a1d0-3e489726e141-kube-api-access-k2xdd\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:15:01.087789 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:01.087662    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a144dbb-03c8-4613-a1d0-3e489726e141-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:15:01.267553 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:01.267515    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"]
Jul 06 21:15:01.269412 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:01.269390    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml"]
Jul 06 21:15:01.474090 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:01.474056    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a144dbb-03c8-4613-a1d0-3e489726e141" path="/var/lib/kubelet/pods/2a144dbb-03c8-4613-a1d0-3e489726e141/volumes"
Jul 06 21:15:47.263871 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.263839    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"]
Jul 06 21:15:47.264455 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.264293    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a144dbb-03c8-4613-a1d0-3e489726e141" containerName="storage-initializer"
Jul 06 21:15:47.264455 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.264311    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a144dbb-03c8-4613-a1d0-3e489726e141" containerName="storage-initializer"
Jul 06 21:15:47.264455 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.264328    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a144dbb-03c8-4613-a1d0-3e489726e141" containerName="main"
Jul 06 21:15:47.264455 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.264336    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a144dbb-03c8-4613-a1d0-3e489726e141" containerName="main"
Jul 06 21:15:47.264455 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.264367    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e753341d-9ed6-4709-9196-2d5d8079e543" containerName="main"
Jul 06 21:15:47.264455 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.264374    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e753341d-9ed6-4709-9196-2d5d8079e543" containerName="main"
Jul 06 21:15:47.264455 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.264390    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e753341d-9ed6-4709-9196-2d5d8079e543" containerName="storage-initializer"
Jul 06 21:15:47.264455 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.264398    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e753341d-9ed6-4709-9196-2d5d8079e543" containerName="storage-initializer"
Jul 06 21:15:47.264879 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.264500    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e753341d-9ed6-4709-9196-2d5d8079e543" containerName="main"
Jul 06 21:15:47.264879 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.264513    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a144dbb-03c8-4613-a1d0-3e489726e141" containerName="main"
Jul 06 21:15:47.267799 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.267776    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.271268 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.271245    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:15:47.271268 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.271256    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:15:47.272258 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.272234    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-lkcz4\""
Jul 06 21:15:47.272258 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.272247    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:15:47.272457 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.272291    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:15:47.272457 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.272238    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 06 21:15:47.283952 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.283931    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"]
Jul 06 21:15:47.371198 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.371167    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.371353 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.371205    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19d65123-e060-4c51-9eb3-a44e98e5cee1-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.371353 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.371244    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.371353 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.371271    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8mh\" (UniqueName: \"kubernetes.io/projected/19d65123-e060-4c51-9eb3-a44e98e5cee1-kube-api-access-ds8mh\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.471823 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.471771    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19d65123-e060-4c51-9eb3-a44e98e5cee1-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.472022 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.471880    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.472022 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.471930    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8mh\" (UniqueName: \"kubernetes.io/projected/19d65123-e060-4c51-9eb3-a44e98e5cee1-kube-api-access-ds8mh\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.472022 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.471990    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.472315 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.472286    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.472394 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.472379    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.474577 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.474560    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 06 21:15:47.480791 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.480775    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:15:47.485140 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.485117    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19d65123-e060-4c51-9eb3-a44e98e5cee1-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.492325 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.492307    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:15:47.501721 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.501700    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8mh\" (UniqueName: \"kubernetes.io/projected/19d65123-e060-4c51-9eb3-a44e98e5cee1-kube-api-access-ds8mh\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.583340 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.583317    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:15:47.593306 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.593287    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-lkcz4\""
Jul 06 21:15:47.601589 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.601573    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:47.743461 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.743435    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"]
Jul 06 21:15:47.746022 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:15:47.745990    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d65123_e060_4c51_9eb3_a44e98e5cee1.slice/crio-f9fae832539476d610f9bbffb359d33f15010ca3c16f3cdf347819c5640bf3d5 WatchSource:0}: Error finding container f9fae832539476d610f9bbffb359d33f15010ca3c16f3cdf347819c5640bf3d5: Status 404 returned error can't find the container with id f9fae832539476d610f9bbffb359d33f15010ca3c16f3cdf347819c5640bf3d5
Jul 06 21:15:47.751756 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:47.751734    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:15:48.150477 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:48.150447    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4" event={"ID":"19d65123-e060-4c51-9eb3-a44e98e5cee1","Type":"ContainerStarted","Data":"fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4"}
Jul 06 21:15:48.150688 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:48.150497    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4" event={"ID":"19d65123-e060-4c51-9eb3-a44e98e5cee1","Type":"ContainerStarted","Data":"f9fae832539476d610f9bbffb359d33f15010ca3c16f3cdf347819c5640bf3d5"}
Jul 06 21:15:49.156879 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:49.156847    2550 generic.go:356] "Generic (PLEG): container finished" podID="19d65123-e060-4c51-9eb3-a44e98e5cee1" containerID="fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4" exitCode=0
Jul 06 21:15:49.157248 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:49.156925    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4" event={"ID":"19d65123-e060-4c51-9eb3-a44e98e5cee1","Type":"ContainerDied","Data":"fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4"}
Jul 06 21:15:50.162883 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:50.162849    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4" event={"ID":"19d65123-e060-4c51-9eb3-a44e98e5cee1","Type":"ContainerStarted","Data":"a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2"}
Jul 06 21:15:50.163274 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:50.162932    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:15:50.195464 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:15:50.195414    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4" podStartSLOduration=2.517811513 podStartE2EDuration="3.195399281s" totalImagesPullingTime="0s" totalInitContainerRuntime="677.587768ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:15:50.193981692 +0000 UTC m=+2463.135285539" watchObservedRunningTime="2026-07-06 21:15:50.195399281 +0000 UTC m=+2463.136703127"
Jul 06 21:16:10.485375 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.485347    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"]
Jul 06 21:16:10.488663 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.488648    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.492286 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.492253    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-mxv24\""
Jul 06 21:16:10.492435 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.492297    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 06 21:16:10.502224 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.502204    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"]
Jul 06 21:16:10.567988 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.567959    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.568140 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.568059    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.568140 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.568097    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszdd\" (UniqueName: \"kubernetes.io/projected/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kube-api-access-rszdd\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.568140 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.568122    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.668510 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.668475    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszdd\" (UniqueName: \"kubernetes.io/projected/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kube-api-access-rszdd\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.668665 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.668520    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.668665 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.668564    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.668665 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.668636    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.669073 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.669053    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.669150 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.669095    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.670955 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.670938    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.680389 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.680368    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszdd\" (UniqueName: \"kubernetes.io/projected/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kube-api-access-rszdd\") pod \"stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.797925 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.797842    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:10.930937 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:10.930910    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"]
Jul 06 21:16:10.932645 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:16:10.932616    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode292f0fc_a6e0_4bf9_9986_a3e673ed650a.slice/crio-94fb4985d87572f8ecb3b8bfd20900e513afa989fa19235282b8c4717ad76cd1 WatchSource:0}: Error finding container 94fb4985d87572f8ecb3b8bfd20900e513afa989fa19235282b8c4717ad76cd1: Status 404 returned error can't find the container with id 94fb4985d87572f8ecb3b8bfd20900e513afa989fa19235282b8c4717ad76cd1
Jul 06 21:16:11.250864 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:11.250823    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s" event={"ID":"e292f0fc-a6e0-4bf9-9986-a3e673ed650a","Type":"ContainerStarted","Data":"8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463"}
Jul 06 21:16:11.251030 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:11.250880    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s" event={"ID":"e292f0fc-a6e0-4bf9-9986-a3e673ed650a","Type":"ContainerStarted","Data":"94fb4985d87572f8ecb3b8bfd20900e513afa989fa19235282b8c4717ad76cd1"}
Jul 06 21:16:12.256829 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:12.256769    2550 generic.go:356] "Generic (PLEG): container finished" podID="e292f0fc-a6e0-4bf9-9986-a3e673ed650a" containerID="8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463" exitCode=0
Jul 06 21:16:12.257231 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:12.256853    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s" event={"ID":"e292f0fc-a6e0-4bf9-9986-a3e673ed650a","Type":"ContainerDied","Data":"8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463"}
Jul 06 21:16:13.263486 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:13.263447    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s" event={"ID":"e292f0fc-a6e0-4bf9-9986-a3e673ed650a","Type":"ContainerStarted","Data":"7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27"}
Jul 06 21:16:13.263883 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:13.263595    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:16:13.286197 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:13.286153    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s" podStartSLOduration=2.52491653 podStartE2EDuration="3.286139205s" totalImagesPullingTime="0s" totalInitContainerRuntime="761.222675ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:16:13.284905377 +0000 UTC m=+2486.226209225" watchObservedRunningTime="2026-07-06 21:16:13.286139205 +0000 UTC m=+2486.227443051"
Jul 06 21:16:21.168376 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:21.168347    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:16:44.269106 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:16:44.269030    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:19:21.675069 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:21.675036    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"]
Jul 06 21:19:21.684481 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:19:21.684440    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:19:21.684481 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:19:21.684481 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:19:21.684481 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2" cmd=["/bin/sleep","15"]
Jul 06 21:19:21.684722 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:21.684496    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4" message="" err=<
Jul 06 21:19:21.684722 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:19:21.684722 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:19:21.684722 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:19:21.684722 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:19:21.684510    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:19:21.684722 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:19:21.684722 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:19:21.684722 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4" podUID="19d65123-e060-4c51-9eb3-a44e98e5cee1" containerName="main" containerID="cri-o://a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2"
Jul 06 21:19:21.684722 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:21.684563    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4" podUID="19d65123-e060-4c51-9eb3-a44e98e5cee1" containerName="main" containerID="cri-o://a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2" gracePeriod=60
Jul 06 21:19:21.939125 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:21.939059    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:19:22.058608 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.058577    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds8mh\" (UniqueName: \"kubernetes.io/projected/19d65123-e060-4c51-9eb3-a44e98e5cee1-kube-api-access-ds8mh\") pod \"19d65123-e060-4c51-9eb3-a44e98e5cee1\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") "
Jul 06 21:19:22.058608 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.058611    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-tokenizer-uds\") pod \"19d65123-e060-4c51-9eb3-a44e98e5cee1\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") "
Jul 06 21:19:22.058847 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.058643    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19d65123-e060-4c51-9eb3-a44e98e5cee1-tls-certs\") pod \"19d65123-e060-4c51-9eb3-a44e98e5cee1\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") "
Jul 06 21:19:22.058847 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.058703    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-kserve-provision-location\") pod \"19d65123-e060-4c51-9eb3-a44e98e5cee1\" (UID: \"19d65123-e060-4c51-9eb3-a44e98e5cee1\") "
Jul 06 21:19:22.058971 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.058851    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "19d65123-e060-4c51-9eb3-a44e98e5cee1" (UID: "19d65123-e060-4c51-9eb3-a44e98e5cee1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:22.059033 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.058972    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:19:22.059475 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.059450    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19d65123-e060-4c51-9eb3-a44e98e5cee1" (UID: "19d65123-e060-4c51-9eb3-a44e98e5cee1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:22.060710 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.060688    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d65123-e060-4c51-9eb3-a44e98e5cee1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19d65123-e060-4c51-9eb3-a44e98e5cee1" (UID: "19d65123-e060-4c51-9eb3-a44e98e5cee1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:19:22.060776 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.060726    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d65123-e060-4c51-9eb3-a44e98e5cee1-kube-api-access-ds8mh" (OuterVolumeSpecName: "kube-api-access-ds8mh") pod "19d65123-e060-4c51-9eb3-a44e98e5cee1" (UID: "19d65123-e060-4c51-9eb3-a44e98e5cee1"). InnerVolumeSpecName "kube-api-access-ds8mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:19:22.091382 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.091352    2550 generic.go:356] "Generic (PLEG): container finished" podID="19d65123-e060-4c51-9eb3-a44e98e5cee1" containerID="a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2" exitCode=0
Jul 06 21:19:22.091485 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.091404    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4" event={"ID":"19d65123-e060-4c51-9eb3-a44e98e5cee1","Type":"ContainerDied","Data":"a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2"}
Jul 06 21:19:22.091485 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.091409    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"
Jul 06 21:19:22.091485 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.091479    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4" event={"ID":"19d65123-e060-4c51-9eb3-a44e98e5cee1","Type":"ContainerDied","Data":"f9fae832539476d610f9bbffb359d33f15010ca3c16f3cdf347819c5640bf3d5"}
Jul 06 21:19:22.091613 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.091506    2550 scope.go:117] "RemoveContainer" containerID="a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2"
Jul 06 21:19:22.100556 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.100538    2550 scope.go:117] "RemoveContainer" containerID="fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4"
Jul 06 21:19:22.107819 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.107791    2550 scope.go:117] "RemoveContainer" containerID="a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2"
Jul 06 21:19:22.108112 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:19:22.108095    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2\": container with ID starting with a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2 not found: ID does not exist" containerID="a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2"
Jul 06 21:19:22.108250 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.108118    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2"} err="failed to get container status \"a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2\": rpc error: code = NotFound desc = could not find container \"a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2\": container with ID starting with a7bf8f1c59ce50f06a3396ed356861d2e0b3ff914c2f230a231319cbd8630fd2 not found: ID does not exist"
Jul 06 21:19:22.108250 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.108143    2550 scope.go:117] "RemoveContainer" containerID="fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4"
Jul 06 21:19:22.108401 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:19:22.108377    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4\": container with ID starting with fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4 not found: ID does not exist" containerID="fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4"
Jul 06 21:19:22.108437 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.108411    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4"} err="failed to get container status \"fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4\": rpc error: code = NotFound desc = could not find container \"fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4\": container with ID starting with fe73e7674ac4807a955e13684c9777e9228e65fd1bee38b3882503cd22e812c4 not found: ID does not exist"
Jul 06 21:19:22.137998 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.137978    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"]
Jul 06 21:19:22.144264 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.144246    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4"]
Jul 06 21:19:22.160324 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.160305    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds8mh\" (UniqueName: \"kubernetes.io/projected/19d65123-e060-4c51-9eb3-a44e98e5cee1-kube-api-access-ds8mh\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:19:22.160408 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.160327    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19d65123-e060-4c51-9eb3-a44e98e5cee1-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:19:22.160408 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:22.160344    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19d65123-e060-4c51-9eb3-a44e98e5cee1-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:19:23.043199 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.043167    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"]
Jul 06 21:19:23.052332 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:19:23.052298    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:19:23.052332 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:19:23.052332 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:19:23.052332 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27" cmd=["/bin/sleep","15"]
Jul 06 21:19:23.052711 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.052344    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s" message="" err=<
Jul 06 21:19:23.052711 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:19:23.052711 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:19:23.052711 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:19:23.052711 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:19:23.052356    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:19:23.052711 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:19:23.052711 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:19:23.052711 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s" podUID="e292f0fc-a6e0-4bf9-9986-a3e673ed650a" containerName="main" containerID="cri-o://7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27"
Jul 06 21:19:23.052711 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.052405    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s" podUID="e292f0fc-a6e0-4bf9-9986-a3e673ed650a" containerName="main" containerID="cri-o://7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27" gracePeriod=60
Jul 06 21:19:23.300732 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.300679    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:19:23.371639 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.371612    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tls-certs\") pod \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") "
Jul 06 21:19:23.371845 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.371668    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tokenizer-uds\") pod \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") "
Jul 06 21:19:23.371845 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.371690    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rszdd\" (UniqueName: \"kubernetes.io/projected/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kube-api-access-rszdd\") pod \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") "
Jul 06 21:19:23.371845 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.371720    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kserve-provision-location\") pod \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\" (UID: \"e292f0fc-a6e0-4bf9-9986-a3e673ed650a\") "
Jul 06 21:19:23.372036 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.371967    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e292f0fc-a6e0-4bf9-9986-a3e673ed650a" (UID: "e292f0fc-a6e0-4bf9-9986-a3e673ed650a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:23.372450 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.372425    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e292f0fc-a6e0-4bf9-9986-a3e673ed650a" (UID: "e292f0fc-a6e0-4bf9-9986-a3e673ed650a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:23.373869 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.373844    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e292f0fc-a6e0-4bf9-9986-a3e673ed650a" (UID: "e292f0fc-a6e0-4bf9-9986-a3e673ed650a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:19:23.373971 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.373914    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kube-api-access-rszdd" (OuterVolumeSpecName: "kube-api-access-rszdd") pod "e292f0fc-a6e0-4bf9-9986-a3e673ed650a" (UID: "e292f0fc-a6e0-4bf9-9986-a3e673ed650a"). InnerVolumeSpecName "kube-api-access-rszdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:19:23.472798 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.472775    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rszdd\" (UniqueName: \"kubernetes.io/projected/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kube-api-access-rszdd\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:19:23.472798 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.472799    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:19:23.472965 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.472821    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:19:23.472965 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.472833    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e292f0fc-a6e0-4bf9-9986-a3e673ed650a-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:19:23.473609 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:23.473588    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19d65123-e060-4c51-9eb3-a44e98e5cee1" path="/var/lib/kubelet/pods/19d65123-e060-4c51-9eb3-a44e98e5cee1/volumes"
Jul 06 21:19:24.101829 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.101777    2550 generic.go:356] "Generic (PLEG): container finished" podID="e292f0fc-a6e0-4bf9-9986-a3e673ed650a" containerID="7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27" exitCode=0
Jul 06 21:19:24.102233 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.101850    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"
Jul 06 21:19:24.102233 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.101853    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s" event={"ID":"e292f0fc-a6e0-4bf9-9986-a3e673ed650a","Type":"ContainerDied","Data":"7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27"}
Jul 06 21:19:24.102233 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.101907    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s" event={"ID":"e292f0fc-a6e0-4bf9-9986-a3e673ed650a","Type":"ContainerDied","Data":"94fb4985d87572f8ecb3b8bfd20900e513afa989fa19235282b8c4717ad76cd1"}
Jul 06 21:19:24.102233 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.101936    2550 scope.go:117] "RemoveContainer" containerID="7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27"
Jul 06 21:19:24.110300 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.110280    2550 scope.go:117] "RemoveContainer" containerID="8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463"
Jul 06 21:19:24.117567 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.117549    2550 scope.go:117] "RemoveContainer" containerID="7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27"
Jul 06 21:19:24.117788 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:19:24.117769    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27\": container with ID starting with 7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27 not found: ID does not exist" containerID="7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27"
Jul 06 21:19:24.117885 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.117801    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27"} err="failed to get container status \"7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27\": rpc error: code = NotFound desc = could not find container \"7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27\": container with ID starting with 7e40e7116ee6e03b8fbea00091a0fb3fc42548f803694a52d33bb9317fda1c27 not found: ID does not exist"
Jul 06 21:19:24.117885 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.117845    2550 scope.go:117] "RemoveContainer" containerID="8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463"
Jul 06 21:19:24.118125 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:19:24.118108    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463\": container with ID starting with 8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463 not found: ID does not exist" containerID="8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463"
Jul 06 21:19:24.118161 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.118130    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463"} err="failed to get container status \"8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463\": rpc error: code = NotFound desc = could not find container \"8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463\": container with ID starting with 8fe6ffdc4f9aed4f37272958d06e21d183f8b11a4cbb09d3f1c6994be9807463 not found: ID does not exist"
Jul 06 21:19:24.135522 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.135498    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"]
Jul 06 21:19:24.142680 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:24.142659    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s"]
Jul 06 21:19:25.473401 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:25.473368    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e292f0fc-a6e0-4bf9-9986-a3e673ed650a" path="/var/lib/kubelet/pods/e292f0fc-a6e0-4bf9-9986-a3e673ed650a/volumes"
Jul 06 21:19:59.539800 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.539764    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"]
Jul 06 21:19:59.540229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.540087    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d65123-e060-4c51-9eb3-a44e98e5cee1" containerName="storage-initializer"
Jul 06 21:19:59.540229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.540098    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d65123-e060-4c51-9eb3-a44e98e5cee1" containerName="storage-initializer"
Jul 06 21:19:59.540229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.540110    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e292f0fc-a6e0-4bf9-9986-a3e673ed650a" containerName="storage-initializer"
Jul 06 21:19:59.540229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.540115    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e292f0fc-a6e0-4bf9-9986-a3e673ed650a" containerName="storage-initializer"
Jul 06 21:19:59.540229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.540125    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d65123-e060-4c51-9eb3-a44e98e5cee1" containerName="main"
Jul 06 21:19:59.540229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.540130    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d65123-e060-4c51-9eb3-a44e98e5cee1" containerName="main"
Jul 06 21:19:59.540229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.540136    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e292f0fc-a6e0-4bf9-9986-a3e673ed650a" containerName="main"
Jul 06 21:19:59.540229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.540142    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e292f0fc-a6e0-4bf9-9986-a3e673ed650a" containerName="main"
Jul 06 21:19:59.540229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.540188    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e292f0fc-a6e0-4bf9-9986-a3e673ed650a" containerName="main"
Jul 06 21:19:59.540229 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.540202    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d65123-e060-4c51-9eb3-a44e98e5cee1" containerName="main"
Jul 06 21:19:59.543230 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.543210    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.546150 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.546128    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:19:59.546276 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.546128    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:19:59.546276 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.546246    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 06 21:19:59.547154 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.547129    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:19:59.547249 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.547169    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-79rfg\""
Jul 06 21:19:59.547249 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.547222    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:19:59.557926 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.557904    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"]
Jul 06 21:19:59.648639 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.648611    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d74908-c659-49e4-881f-812e7b8d1ec0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.648800 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.648662    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.648800 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.648692    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdr8t\" (UniqueName: \"kubernetes.io/projected/32d74908-c659-49e4-881f-812e7b8d1ec0-kube-api-access-hdr8t\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.648800 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.648729    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.749145 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.749112    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d74908-c659-49e4-881f-812e7b8d1ec0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.749308 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.749165    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.749373 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.749351    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdr8t\" (UniqueName: \"kubernetes.io/projected/32d74908-c659-49e4-881f-812e7b8d1ec0-kube-api-access-hdr8t\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.749434 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.749403    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.749489 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.749452    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.749735 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.749715    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.751369 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.751352    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d74908-c659-49e4-881f-812e7b8d1ec0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.759167 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.759140    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdr8t\" (UniqueName: \"kubernetes.io/projected/32d74908-c659-49e4-881f-812e7b8d1ec0-kube-api-access-hdr8t\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.852925 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.852901    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:19:59.990381 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:19:59.990360    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"]
Jul 06 21:19:59.992627 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:19:59.992598    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d74908_c659_49e4_881f_812e7b8d1ec0.slice/crio-f4d2d1963c6e58cffd86f353c6e39a51cd705ed763bcd193c1e991758d5338c2 WatchSource:0}: Error finding container f4d2d1963c6e58cffd86f353c6e39a51cd705ed763bcd193c1e991758d5338c2: Status 404 returned error can't find the container with id f4d2d1963c6e58cffd86f353c6e39a51cd705ed763bcd193c1e991758d5338c2
Jul 06 21:20:00.256535 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:20:00.256448    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd" event={"ID":"32d74908-c659-49e4-881f-812e7b8d1ec0","Type":"ContainerStarted","Data":"3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee"}
Jul 06 21:20:00.256535 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:20:00.256505    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd" event={"ID":"32d74908-c659-49e4-881f-812e7b8d1ec0","Type":"ContainerStarted","Data":"f4d2d1963c6e58cffd86f353c6e39a51cd705ed763bcd193c1e991758d5338c2"}
Jul 06 21:20:01.261982 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:20:01.261946    2550 generic.go:356] "Generic (PLEG): container finished" podID="32d74908-c659-49e4-881f-812e7b8d1ec0" containerID="3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee" exitCode=0
Jul 06 21:20:01.262374 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:20:01.262033    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd" event={"ID":"32d74908-c659-49e4-881f-812e7b8d1ec0","Type":"ContainerDied","Data":"3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee"}
Jul 06 21:20:02.268332 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:20:02.268299    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd" event={"ID":"32d74908-c659-49e4-881f-812e7b8d1ec0","Type":"ContainerStarted","Data":"f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812"}
Jul 06 21:20:02.268733 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:20:02.268359    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:20:02.290889 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:20:02.290837    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd" podStartSLOduration=2.6316646930000003 podStartE2EDuration="3.290802615s" totalImagesPullingTime="0s" totalInitContainerRuntime="659.137922ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:20:02.28985249 +0000 UTC m=+2715.231156338" watchObservedRunningTime="2026-07-06 21:20:02.290802615 +0000 UTC m=+2715.232106461"
Jul 06 21:20:33.274754 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:20:33.274720    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:41:05.115962 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.115925    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"]
Jul 06 21:41:05.125472 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:41:05.125438    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:41:05.125472 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:41:05.125472 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:41:05.125472 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812" cmd=["/bin/sleep","15"]
Jul 06 21:41:05.125754 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.125486    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd" message="" err=<
Jul 06 21:41:05.125754 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:41:05.125754 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:41:05.125754 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:41:05.125754 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:41:05.125498    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:41:05.125754 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:41:05.125754 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:41:05.125754 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd" podUID="32d74908-c659-49e4-881f-812e7b8d1ec0" containerName="main" containerID="cri-o://f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812"
Jul 06 21:41:05.125754 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.125548    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd" podUID="32d74908-c659-49e4-881f-812e7b8d1ec0" containerName="main" containerID="cri-o://f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812" gracePeriod=60
Jul 06 21:41:05.376871 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.376801    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:41:05.463130 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.463099    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdr8t\" (UniqueName: \"kubernetes.io/projected/32d74908-c659-49e4-881f-812e7b8d1ec0-kube-api-access-hdr8t\") pod \"32d74908-c659-49e4-881f-812e7b8d1ec0\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") "
Jul 06 21:41:05.463280 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.463144    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d74908-c659-49e4-881f-812e7b8d1ec0-tls-certs\") pod \"32d74908-c659-49e4-881f-812e7b8d1ec0\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") "
Jul 06 21:41:05.463331 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.463290    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-tokenizer-uds\") pod \"32d74908-c659-49e4-881f-812e7b8d1ec0\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") "
Jul 06 21:41:05.463374 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.463337    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-kserve-provision-location\") pod \"32d74908-c659-49e4-881f-812e7b8d1ec0\" (UID: \"32d74908-c659-49e4-881f-812e7b8d1ec0\") "
Jul 06 21:41:05.463510 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.463492    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "32d74908-c659-49e4-881f-812e7b8d1ec0" (UID: "32d74908-c659-49e4-881f-812e7b8d1ec0"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:05.463628 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.463615    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:41:05.464039 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.464017    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32d74908-c659-49e4-881f-812e7b8d1ec0" (UID: "32d74908-c659-49e4-881f-812e7b8d1ec0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:05.465211 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.465187    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d74908-c659-49e4-881f-812e7b8d1ec0-kube-api-access-hdr8t" (OuterVolumeSpecName: "kube-api-access-hdr8t") pod "32d74908-c659-49e4-881f-812e7b8d1ec0" (UID: "32d74908-c659-49e4-881f-812e7b8d1ec0"). InnerVolumeSpecName "kube-api-access-hdr8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:41:05.465298 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.465239    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d74908-c659-49e4-881f-812e7b8d1ec0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32d74908-c659-49e4-881f-812e7b8d1ec0" (UID: "32d74908-c659-49e4-881f-812e7b8d1ec0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:41:05.564045 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.564012    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d74908-c659-49e4-881f-812e7b8d1ec0-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:41:05.564045 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.564039    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32d74908-c659-49e4-881f-812e7b8d1ec0-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:41:05.564045 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.564048    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdr8t\" (UniqueName: \"kubernetes.io/projected/32d74908-c659-49e4-881f-812e7b8d1ec0-kube-api-access-hdr8t\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:41:05.733612 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.733523    2550 generic.go:356] "Generic (PLEG): container finished" podID="32d74908-c659-49e4-881f-812e7b8d1ec0" containerID="f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812" exitCode=0
Jul 06 21:41:05.733612 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.733568    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd" event={"ID":"32d74908-c659-49e4-881f-812e7b8d1ec0","Type":"ContainerDied","Data":"f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812"}
Jul 06 21:41:05.733612 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.733578    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"
Jul 06 21:41:05.733878 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.733620    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd" event={"ID":"32d74908-c659-49e4-881f-812e7b8d1ec0","Type":"ContainerDied","Data":"f4d2d1963c6e58cffd86f353c6e39a51cd705ed763bcd193c1e991758d5338c2"}
Jul 06 21:41:05.733878 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.733646    2550 scope.go:117] "RemoveContainer" containerID="f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812"
Jul 06 21:41:05.742056 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.742037    2550 scope.go:117] "RemoveContainer" containerID="3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee"
Jul 06 21:41:05.749632 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.749507    2550 scope.go:117] "RemoveContainer" containerID="f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812"
Jul 06 21:41:05.749881 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:41:05.749787    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812\": container with ID starting with f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812 not found: ID does not exist" containerID="f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812"
Jul 06 21:41:05.749881 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.749838    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812"} err="failed to get container status \"f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812\": rpc error: code = NotFound desc = could not find container \"f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812\": container with ID starting with f11a042f11807f91fe4656365a69627eb9a6c5c1748eb55e3b9832c24595e812 not found: ID does not exist"
Jul 06 21:41:05.749881 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.749860    2550 scope.go:117] "RemoveContainer" containerID="3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee"
Jul 06 21:41:05.750165 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:41:05.750144    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee\": container with ID starting with 3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee not found: ID does not exist" containerID="3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee"
Jul 06 21:41:05.750219 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.750170    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee"} err="failed to get container status \"3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee\": rpc error: code = NotFound desc = could not find container \"3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee\": container with ID starting with 3c5022c5414f815c8b794ac451ae354b70fbf37e838c752e27c85b9808341bee not found: ID does not exist"
Jul 06 21:41:05.751642 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.751623    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"]
Jul 06 21:41:05.753001 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:05.752981    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd"]
Jul 06 21:41:07.473767 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:07.473737    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d74908-c659-49e4-881f-812e7b8d1ec0" path="/var/lib/kubelet/pods/32d74908-c659-49e4-881f-812e7b8d1ec0/volumes"
Jul 06 21:41:25.897867 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.897836    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"]
Jul 06 21:41:25.898275 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.898141    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d74908-c659-49e4-881f-812e7b8d1ec0" containerName="storage-initializer"
Jul 06 21:41:25.898275 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.898152    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d74908-c659-49e4-881f-812e7b8d1ec0" containerName="storage-initializer"
Jul 06 21:41:25.898275 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.898172    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d74908-c659-49e4-881f-812e7b8d1ec0" containerName="main"
Jul 06 21:41:25.898275 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.898178    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d74908-c659-49e4-881f-812e7b8d1ec0" containerName="main"
Jul 06 21:41:25.898275 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.898228    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d74908-c659-49e4-881f-812e7b8d1ec0" containerName="main"
Jul 06 21:41:25.909559 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.909528    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:25.912246 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.912224    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-7gh9t\""
Jul 06 21:41:25.912995 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.912974    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:41:25.913154 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.913013    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 06 21:41:25.913154 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.913066    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:41:25.914035 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:25.914016    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"]
Jul 06 21:41:26.022641 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.022609    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.022641 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.022645    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a339f3e3-5a39-418e-96ba-1436872939a0-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.022877 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.022668    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.022877 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.022701    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.022877 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.022761    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5pd\" (UniqueName: \"kubernetes.io/projected/a339f3e3-5a39-418e-96ba-1436872939a0-kube-api-access-kd5pd\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.022877 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.022795    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a339f3e3-5a39-418e-96ba-1436872939a0-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.022877 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.022851    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a339f3e3-5a39-418e-96ba-1436872939a0-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.022877 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.022872    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.023103 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.022952    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.123600 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.123567    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.123772 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.123606    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.123772 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.123637    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a339f3e3-5a39-418e-96ba-1436872939a0-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.123772 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.123666    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.123772 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.123698    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.123772 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.123726    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5pd\" (UniqueName: \"kubernetes.io/projected/a339f3e3-5a39-418e-96ba-1436872939a0-kube-api-access-kd5pd\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.124077 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.123838    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a339f3e3-5a39-418e-96ba-1436872939a0-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.124077 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.123900    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a339f3e3-5a39-418e-96ba-1436872939a0-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.124077 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.123935    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.124077 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.124024    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.124077 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.124066    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.124309 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.124132    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.124309 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.124281    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.124429 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.124410    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a339f3e3-5a39-418e-96ba-1436872939a0-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.126069 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.126041    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a339f3e3-5a39-418e-96ba-1436872939a0-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.126194 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.126178    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a339f3e3-5a39-418e-96ba-1436872939a0-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.131764 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.131743    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a339f3e3-5a39-418e-96ba-1436872939a0-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.132089 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.132068    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5pd\" (UniqueName: \"kubernetes.io/projected/a339f3e3-5a39-418e-96ba-1436872939a0-kube-api-access-kd5pd\") pod \"router-gateway-2-openshift-default-54c789bdc6-22wcp\" (UID: \"a339f3e3-5a39-418e-96ba-1436872939a0\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.220635 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.220584    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:26.358053 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.357971    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"]
Jul 06 21:41:26.362652 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:41:26.362624    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda339f3e3_5a39_418e_96ba_1436872939a0.slice/crio-d344a5d6d69ac6284b532eeea993c24b15d9b61cdccfb15b7b8183cd1028877e WatchSource:0}: Error finding container d344a5d6d69ac6284b532eeea993c24b15d9b61cdccfb15b7b8183cd1028877e: Status 404 returned error can't find the container with id d344a5d6d69ac6284b532eeea993c24b15d9b61cdccfb15b7b8183cd1028877e
Jul 06 21:41:26.364591 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.364571    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:41:26.825500 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:26.825466    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp" event={"ID":"a339f3e3-5a39-418e-96ba-1436872939a0","Type":"ContainerStarted","Data":"d344a5d6d69ac6284b532eeea993c24b15d9b61cdccfb15b7b8183cd1028877e"}
Jul 06 21:41:28.839583 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:28.839548    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:41:28.839890 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:28.839621    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:41:28.839890 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:28.839649    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:41:29.840838 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:29.840726    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp" event={"ID":"a339f3e3-5a39-418e-96ba-1436872939a0","Type":"ContainerStarted","Data":"cc332a8efae41117db74bda7faf8fbf5cd4c72249d7205fce07cce37454b6d97"}
Jul 06 21:41:29.864307 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:29.864266    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp" podStartSLOduration=2.389641176 podStartE2EDuration="4.864250252s" totalImagesPullingTime="2.474609076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:41:29.862579752 +0000 UTC m=+4002.803883600" watchObservedRunningTime="2026-07-06 21:41:29.864250252 +0000 UTC m=+4002.805554113"
Jul 06 21:41:30.223009 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:30.222911    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:30.223186 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:30.223041    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp" podUID="a339f3e3-5a39-418e-96ba-1436872939a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 06 21:41:31.221330 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:31.221282    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp" podUID="a339f3e3-5a39-418e-96ba-1436872939a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 06 21:41:32.221451 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:32.221400    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp" podUID="a339f3e3-5a39-418e-96ba-1436872939a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 06 21:41:33.221866 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:33.221796    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp" podUID="a339f3e3-5a39-418e-96ba-1436872939a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 06 21:41:34.221264 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:34.221216    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp" podUID="a339f3e3-5a39-418e-96ba-1436872939a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 06 21:41:35.224714 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:35.224685    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:35.225130 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:35.225056    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:35.225772 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:35.225753    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp"
Jul 06 21:41:45.245459 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.245425    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"]
Jul 06 21:41:45.251304 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.251282    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.253912 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.253881    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 06 21:41:45.254856 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.254831    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:41:45.254964 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.254868    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-pktcg\""
Jul 06 21:41:45.254964 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.254869    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:41:45.261494 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.261472    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"]
Jul 06 21:41:45.387507 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.387479    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56b0110-8974-452b-9012-126a6252a7c6-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.387655 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.387511    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.387655 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.387589    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpdh\" (UniqueName: \"kubernetes.io/projected/e56b0110-8974-452b-9012-126a6252a7c6-kube-api-access-jmpdh\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.387731 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.387662    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.488344 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.488309    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpdh\" (UniqueName: \"kubernetes.io/projected/e56b0110-8974-452b-9012-126a6252a7c6-kube-api-access-jmpdh\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.488534 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.488364    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.488534 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.488408    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56b0110-8974-452b-9012-126a6252a7c6-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.488534 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.488433    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.488719 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.488690    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.488837 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.488796    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.490762 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.490745    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56b0110-8974-452b-9012-126a6252a7c6-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.498147 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.498089    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpdh\" (UniqueName: \"kubernetes.io/projected/e56b0110-8974-452b-9012-126a6252a7c6-kube-api-access-jmpdh\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.562080 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.562048    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:45.714589 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.714543    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"]
Jul 06 21:41:45.716867 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:41:45.716838    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56b0110_8974_452b_9012_126a6252a7c6.slice/crio-c56d1e4baf995eae8a9d8d3d007f92779dfcfa68b48bbbab7450b66f15393984 WatchSource:0}: Error finding container c56d1e4baf995eae8a9d8d3d007f92779dfcfa68b48bbbab7450b66f15393984: Status 404 returned error can't find the container with id c56d1e4baf995eae8a9d8d3d007f92779dfcfa68b48bbbab7450b66f15393984
Jul 06 21:41:45.911540 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.911504    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8" event={"ID":"e56b0110-8974-452b-9012-126a6252a7c6","Type":"ContainerStarted","Data":"66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50"}
Jul 06 21:41:45.911733 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:45.911564    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8" event={"ID":"e56b0110-8974-452b-9012-126a6252a7c6","Type":"ContainerStarted","Data":"c56d1e4baf995eae8a9d8d3d007f92779dfcfa68b48bbbab7450b66f15393984"}
Jul 06 21:41:46.917100 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:46.917063    2550 generic.go:356] "Generic (PLEG): container finished" podID="e56b0110-8974-452b-9012-126a6252a7c6" containerID="66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50" exitCode=0
Jul 06 21:41:46.917569 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:46.917107    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8" event={"ID":"e56b0110-8974-452b-9012-126a6252a7c6","Type":"ContainerDied","Data":"66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50"}
Jul 06 21:41:47.922472 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:47.922441    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8" event={"ID":"e56b0110-8974-452b-9012-126a6252a7c6","Type":"ContainerStarted","Data":"ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6"}
Jul 06 21:41:47.922782 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:47.922531    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:41:47.943803 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:41:47.943761    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8" podStartSLOduration=2.310528912 podStartE2EDuration="2.943748188s" totalImagesPullingTime="0s" totalInitContainerRuntime="633.219276ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:41:47.943504886 +0000 UTC m=+4020.884808746" watchObservedRunningTime="2026-07-06 21:41:47.943748188 +0000 UTC m=+4020.885052037"
Jul 06 21:42:18.930127 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:42:18.930040    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:44:44.607053 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.607021    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"]
Jul 06 21:44:44.616063 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:44:44.616025    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:44:44.616063 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:44:44.616063 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:44:44.616063 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6" cmd=["/bin/sleep","15"]
Jul 06 21:44:44.616375 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.616083    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8" message="" err=<
Jul 06 21:44:44.616375 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:44:44.616375 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:44:44.616375 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:44:44.616375 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:44:44.616098    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:44:44.616375 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:44:44.616375 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:44:44.616375 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8" podUID="e56b0110-8974-452b-9012-126a6252a7c6" containerName="main" containerID="cri-o://ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6"
Jul 06 21:44:44.616375 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.616156    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8" podUID="e56b0110-8974-452b-9012-126a6252a7c6" containerName="main" containerID="cri-o://ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6" gracePeriod=60
Jul 06 21:44:44.878335 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.878308    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:44:44.991888 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.991858    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmpdh\" (UniqueName: \"kubernetes.io/projected/e56b0110-8974-452b-9012-126a6252a7c6-kube-api-access-jmpdh\") pod \"e56b0110-8974-452b-9012-126a6252a7c6\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") "
Jul 06 21:44:44.992057 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.991920    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-kserve-provision-location\") pod \"e56b0110-8974-452b-9012-126a6252a7c6\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") "
Jul 06 21:44:44.992057 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.991941    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-tokenizer-uds\") pod \"e56b0110-8974-452b-9012-126a6252a7c6\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") "
Jul 06 21:44:44.992057 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.991968    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56b0110-8974-452b-9012-126a6252a7c6-tls-certs\") pod \"e56b0110-8974-452b-9012-126a6252a7c6\" (UID: \"e56b0110-8974-452b-9012-126a6252a7c6\") "
Jul 06 21:44:44.992225 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.992183    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e56b0110-8974-452b-9012-126a6252a7c6" (UID: "e56b0110-8974-452b-9012-126a6252a7c6"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:44:44.992546 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.992515    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e56b0110-8974-452b-9012-126a6252a7c6" (UID: "e56b0110-8974-452b-9012-126a6252a7c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:44:44.994071 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.994047    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56b0110-8974-452b-9012-126a6252a7c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e56b0110-8974-452b-9012-126a6252a7c6" (UID: "e56b0110-8974-452b-9012-126a6252a7c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:44:44.994191 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:44.994070    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56b0110-8974-452b-9012-126a6252a7c6-kube-api-access-jmpdh" (OuterVolumeSpecName: "kube-api-access-jmpdh") pod "e56b0110-8974-452b-9012-126a6252a7c6" (UID: "e56b0110-8974-452b-9012-126a6252a7c6"). InnerVolumeSpecName "kube-api-access-jmpdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:44:45.092821 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.092787    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:44:45.092821 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.092824    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e56b0110-8974-452b-9012-126a6252a7c6-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:44:45.093079 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.092835    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56b0110-8974-452b-9012-126a6252a7c6-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:44:45.093079 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.092843    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpdh\" (UniqueName: \"kubernetes.io/projected/e56b0110-8974-452b-9012-126a6252a7c6-kube-api-access-jmpdh\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:44:45.694885 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.694852    2550 generic.go:356] "Generic (PLEG): container finished" podID="e56b0110-8974-452b-9012-126a6252a7c6" containerID="ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6" exitCode=0
Jul 06 21:44:45.695359 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.694912    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"
Jul 06 21:44:45.695359 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.694936    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8" event={"ID":"e56b0110-8974-452b-9012-126a6252a7c6","Type":"ContainerDied","Data":"ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6"}
Jul 06 21:44:45.695359 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.694993    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8" event={"ID":"e56b0110-8974-452b-9012-126a6252a7c6","Type":"ContainerDied","Data":"c56d1e4baf995eae8a9d8d3d007f92779dfcfa68b48bbbab7450b66f15393984"}
Jul 06 21:44:45.695359 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.695014    2550 scope.go:117] "RemoveContainer" containerID="ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6"
Jul 06 21:44:45.705306 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.705290    2550 scope.go:117] "RemoveContainer" containerID="66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50"
Jul 06 21:44:45.712979 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.712960    2550 scope.go:117] "RemoveContainer" containerID="ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6"
Jul 06 21:44:45.713242 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:44:45.713226    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6\": container with ID starting with ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6 not found: ID does not exist" containerID="ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6"
Jul 06 21:44:45.713295 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.713253    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6"} err="failed to get container status \"ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6\": rpc error: code = NotFound desc = could not find container \"ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6\": container with ID starting with ed4b7b1167a333d487fd0c4f0d5991a33d04d3084cef7c107786c25ca83fbac6 not found: ID does not exist"
Jul 06 21:44:45.713295 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.713270    2550 scope.go:117] "RemoveContainer" containerID="66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50"
Jul 06 21:44:45.713529 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:44:45.713508    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50\": container with ID starting with 66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50 not found: ID does not exist" containerID="66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50"
Jul 06 21:44:45.713575 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.713540    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50"} err="failed to get container status \"66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50\": rpc error: code = NotFound desc = could not find container \"66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50\": container with ID starting with 66d86f0969a7876b7fde7fbd695ceb2115252fe5c8a8378a78d8ee3da1f05c50 not found: ID does not exist"
Jul 06 21:44:45.714339 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.714323    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"]
Jul 06 21:44:45.718388 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:45.718369    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8"]
Jul 06 21:44:47.473468 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:44:47.473434    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e56b0110-8974-452b-9012-126a6252a7c6" path="/var/lib/kubelet/pods/e56b0110-8974-452b-9012-126a6252a7c6/volumes"
Jul 06 21:52:36.679182 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.679150    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"]
Jul 06 21:52:36.679791 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.679494    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56b0110-8974-452b-9012-126a6252a7c6" containerName="storage-initializer"
Jul 06 21:52:36.679791 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.679506    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56b0110-8974-452b-9012-126a6252a7c6" containerName="storage-initializer"
Jul 06 21:52:36.679791 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.679528    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56b0110-8974-452b-9012-126a6252a7c6" containerName="main"
Jul 06 21:52:36.679791 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.679533    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56b0110-8974-452b-9012-126a6252a7c6" containerName="main"
Jul 06 21:52:36.679791 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.679590    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56b0110-8974-452b-9012-126a6252a7c6" containerName="main"
Jul 06 21:52:36.682539 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.682520    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.685126 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.685101    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:52:36.685226 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.685112    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-5bqjv\""
Jul 06 21:52:36.686046 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.686029    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:52:36.686152 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.686070    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 06 21:52:36.692647 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.692627    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"]
Jul 06 21:52:36.827408 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.827376    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.827606 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.827439    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxnr2\" (UniqueName: \"kubernetes.io/projected/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kube-api-access-jxnr2\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.827606 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.827511    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.827606 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.827561    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.928309 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.928275    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.928505 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.928339    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxnr2\" (UniqueName: \"kubernetes.io/projected/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kube-api-access-jxnr2\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.928505 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.928373    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.928505 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.928407    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.928788 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.928769    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.928879 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.928832    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.930800 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.930735    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.937032 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.937011    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxnr2\" (UniqueName: \"kubernetes.io/projected/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kube-api-access-jxnr2\") pod \"tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:36.993154 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:36.993126    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:37.128895 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:37.128869    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"]
Jul 06 21:52:37.131297 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:52:37.131275    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c31a6e8_92a5_48b3_8491_f1350bd8dbcb.slice/crio-d97d449a496b8ecfaaa250f26ad5dfe8566b929ee00dd0e66e0419182ac981c3 WatchSource:0}: Error finding container d97d449a496b8ecfaaa250f26ad5dfe8566b929ee00dd0e66e0419182ac981c3: Status 404 returned error can't find the container with id d97d449a496b8ecfaaa250f26ad5dfe8566b929ee00dd0e66e0419182ac981c3
Jul 06 21:52:37.741997 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:37.741952    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb" event={"ID":"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb","Type":"ContainerStarted","Data":"ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021"}
Jul 06 21:52:37.742451 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:37.742014    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb" event={"ID":"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb","Type":"ContainerStarted","Data":"d97d449a496b8ecfaaa250f26ad5dfe8566b929ee00dd0e66e0419182ac981c3"}
Jul 06 21:52:38.747356 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:38.747318    2550 generic.go:356] "Generic (PLEG): container finished" podID="9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" containerID="ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021" exitCode=0
Jul 06 21:52:38.747768 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:38.747387    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb" event={"ID":"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb","Type":"ContainerDied","Data":"ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021"}
Jul 06 21:52:39.753406 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:39.753370    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb" event={"ID":"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb","Type":"ContainerStarted","Data":"d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579"}
Jul 06 21:52:39.753798 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:39.753524    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:52:39.775583 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:52:39.775525    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb" podStartSLOduration=3.140540231 podStartE2EDuration="3.775508988s" totalImagesPullingTime="0s" totalInitContainerRuntime="634.968757ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:52:39.773623073 +0000 UTC m=+4672.714926899" watchObservedRunningTime="2026-07-06 21:52:39.775508988 +0000 UTC m=+4672.716812834"
Jul 06 21:53:10.758717 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:53:10.758688    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:54:06.679892 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.679854    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"]
Jul 06 21:54:06.683332 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.683313    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.685977 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.685956    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 06 21:54:06.687003 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.686985    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-2zwdj\""
Jul 06 21:54:06.695895 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.695874    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"]
Jul 06 21:54:06.759094 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.759067    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsh5z\" (UniqueName: \"kubernetes.io/projected/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kube-api-access-wsh5z\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.759241 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.759099    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.759241 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.759171    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.759241 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.759203    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.859933 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.859898    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.860098 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.859944    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.860098 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.860001    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsh5z\" (UniqueName: \"kubernetes.io/projected/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kube-api-access-wsh5z\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.860098 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.860028    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.860361 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.860339    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.860405 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.860353    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.862430 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.862410    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.873605 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.873582    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsh5z\" (UniqueName: \"kubernetes.io/projected/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kube-api-access-wsh5z\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:06.993307 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:06.993226    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:07.125011 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:07.124986    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"]
Jul 06 21:54:07.127087 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:54:07.127058    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f488f52_c30a_487a_b5c0_72f333a2f8b8.slice/crio-a69cb444b2d008d6b157d4741578abc881a2ce778df5a633fb4d802d36f642af WatchSource:0}: Error finding container a69cb444b2d008d6b157d4741578abc881a2ce778df5a633fb4d802d36f642af: Status 404 returned error can't find the container with id a69cb444b2d008d6b157d4741578abc881a2ce778df5a633fb4d802d36f642af
Jul 06 21:54:07.138320 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:07.138297    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz" event={"ID":"0f488f52-c30a-487a-b5c0-72f333a2f8b8","Type":"ContainerStarted","Data":"a69cb444b2d008d6b157d4741578abc881a2ce778df5a633fb4d802d36f642af"}
Jul 06 21:54:08.143768 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:08.143677    2550 generic.go:356] "Generic (PLEG): container finished" podID="0f488f52-c30a-487a-b5c0-72f333a2f8b8" containerID="c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b" exitCode=0
Jul 06 21:54:08.143768 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:08.143722    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz" event={"ID":"0f488f52-c30a-487a-b5c0-72f333a2f8b8","Type":"ContainerDied","Data":"c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b"}
Jul 06 21:54:09.150313 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:09.150233    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz" event={"ID":"0f488f52-c30a-487a-b5c0-72f333a2f8b8","Type":"ContainerStarted","Data":"13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0"}
Jul 06 21:54:09.150708 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:09.150352    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:09.171727 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:09.171684    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz" podStartSLOduration=2.540544578 podStartE2EDuration="3.171671289s" totalImagesPullingTime="0s" totalInitContainerRuntime="631.126711ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:54:09.170916157 +0000 UTC m=+4762.112220006" watchObservedRunningTime="2026-07-06 21:54:09.171671289 +0000 UTC m=+4762.112975167"
Jul 06 21:54:40.156270 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:40.156240    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:54:41.965956 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:41.965922    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"]
Jul 06 21:54:41.975836 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:54:41.975782    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:54:41.975836 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:54:41.975836 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:54:41.975836 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579" cmd=["/bin/sleep","15"]
Jul 06 21:54:41.976120 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:41.975846    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb" message="" err=<
Jul 06 21:54:41.976120 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:54:41.976120 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:54:41.976120 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:54:41.976120 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:54:41.975859    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:54:41.976120 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:54:41.976120 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:54:41.976120 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb" podUID="9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" containerName="main" containerID="cri-o://d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579"
Jul 06 21:54:41.976120 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:41.975898    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb" podUID="9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" containerName="main" containerID="cri-o://d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579" gracePeriod=60
Jul 06 21:54:42.230465 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.230399    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:54:42.294451 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.294415    2550 generic.go:356] "Generic (PLEG): container finished" podID="9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" containerID="d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579" exitCode=0
Jul 06 21:54:42.294602 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.294461    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"
Jul 06 21:54:42.294602 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.294495    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb" event={"ID":"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb","Type":"ContainerDied","Data":"d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579"}
Jul 06 21:54:42.294602 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.294553    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb" event={"ID":"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb","Type":"ContainerDied","Data":"d97d449a496b8ecfaaa250f26ad5dfe8566b929ee00dd0e66e0419182ac981c3"}
Jul 06 21:54:42.294602 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.294574    2550 scope.go:117] "RemoveContainer" containerID="d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579"
Jul 06 21:54:42.302778 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.302761    2550 scope.go:117] "RemoveContainer" containerID="ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021"
Jul 06 21:54:42.309784 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.309758    2550 scope.go:117] "RemoveContainer" containerID="d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579"
Jul 06 21:54:42.310039 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:54:42.310022    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579\": container with ID starting with d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579 not found: ID does not exist" containerID="d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579"
Jul 06 21:54:42.310090 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.310049    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579"} err="failed to get container status \"d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579\": rpc error: code = NotFound desc = could not find container \"d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579\": container with ID starting with d8a02ee019668ae97e265fcd82d5baca192590f91f909443568429393fe43579 not found: ID does not exist"
Jul 06 21:54:42.310090 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.310065    2550 scope.go:117] "RemoveContainer" containerID="ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021"
Jul 06 21:54:42.310312 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:54:42.310298    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021\": container with ID starting with ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021 not found: ID does not exist" containerID="ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021"
Jul 06 21:54:42.310352 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.310316    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021"} err="failed to get container status \"ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021\": rpc error: code = NotFound desc = could not find container \"ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021\": container with ID starting with ced53075f743337dd87569e255bc0985c61853f295f0ed499d3db551177fa021 not found: ID does not exist"
Jul 06 21:54:42.359492 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.359466    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tokenizer-uds\") pod \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") "
Jul 06 21:54:42.359633 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.359507    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tls-certs\") pod \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") "
Jul 06 21:54:42.359633 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.359544    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kserve-provision-location\") pod \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") "
Jul 06 21:54:42.359633 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.359607    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxnr2\" (UniqueName: \"kubernetes.io/projected/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kube-api-access-jxnr2\") pod \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\" (UID: \"9c31a6e8-92a5-48b3-8491-f1350bd8dbcb\") "
Jul 06 21:54:42.359834 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.359692    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" (UID: "9c31a6e8-92a5-48b3-8491-f1350bd8dbcb"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:54:42.359834 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.359784    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:54:42.360202 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.360179    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" (UID: "9c31a6e8-92a5-48b3-8491-f1350bd8dbcb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:54:42.361527 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.361506    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" (UID: "9c31a6e8-92a5-48b3-8491-f1350bd8dbcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:54:42.361637 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.361614    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kube-api-access-jxnr2" (OuterVolumeSpecName: "kube-api-access-jxnr2") pod "9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" (UID: "9c31a6e8-92a5-48b3-8491-f1350bd8dbcb"). InnerVolumeSpecName "kube-api-access-jxnr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:54:42.460733 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.460697    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxnr2\" (UniqueName: \"kubernetes.io/projected/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kube-api-access-jxnr2\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:54:42.460733 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.460728    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:54:42.460733 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.460739    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:54:42.615656 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.615625    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"]
Jul 06 21:54:42.619385 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:42.619361    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb"]
Jul 06 21:54:43.473679 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:54:43.473647    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" path="/var/lib/kubelet/pods/9c31a6e8-92a5-48b3-8491-f1350bd8dbcb/volumes"
Jul 06 21:55:45.788212 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.788136    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"]
Jul 06 21:55:45.788714 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.788508    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" containerName="main"
Jul 06 21:55:45.788714 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.788521    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" containerName="main"
Jul 06 21:55:45.788714 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.788535    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" containerName="storage-initializer"
Jul 06 21:55:45.788714 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.788541    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" containerName="storage-initializer"
Jul 06 21:55:45.788714 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.788648    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c31a6e8-92a5-48b3-8491-f1350bd8dbcb" containerName="main"
Jul 06 21:55:45.791129 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.791111    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.793709 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.793686    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 06 21:55:45.794642 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.794624    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-rzmzw\""
Jul 06 21:55:45.807113 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.807063    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"]
Jul 06 21:55:45.878927 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.878893    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.878927 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.878929    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcvng\" (UniqueName: \"kubernetes.io/projected/8ea769ee-754a-4c3b-b34e-a04259907993-kube-api-access-rcvng\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.879131 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.878950    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea769ee-754a-4c3b-b34e-a04259907993-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.879131 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.879046    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.980423 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.980393    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.980580 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.980429    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcvng\" (UniqueName: \"kubernetes.io/projected/8ea769ee-754a-4c3b-b34e-a04259907993-kube-api-access-rcvng\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.980580 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.980450    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea769ee-754a-4c3b-b34e-a04259907993-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.980580 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.980493    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.980797 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.980775    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.980894 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.980878    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.982873 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.982856    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea769ee-754a-4c3b-b34e-a04259907993-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:45.989557 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:45.989535    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcvng\" (UniqueName: \"kubernetes.io/projected/8ea769ee-754a-4c3b-b34e-a04259907993-kube-api-access-rcvng\") pod \"prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:46.100356 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:46.100331    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:46.231362 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:46.231338    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"]
Jul 06 21:55:46.233455 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:55:46.233420    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea769ee_754a_4c3b_b34e_a04259907993.slice/crio-900b9f5d7ce00a9e995f1149f11a68b3db3b4c8d28045756ff12dcd986e0dce6 WatchSource:0}: Error finding container 900b9f5d7ce00a9e995f1149f11a68b3db3b4c8d28045756ff12dcd986e0dce6: Status 404 returned error can't find the container with id 900b9f5d7ce00a9e995f1149f11a68b3db3b4c8d28045756ff12dcd986e0dce6
Jul 06 21:55:46.576226 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:46.576182    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" event={"ID":"8ea769ee-754a-4c3b-b34e-a04259907993","Type":"ContainerStarted","Data":"a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d"}
Jul 06 21:55:46.576439 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:46.576241    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" event={"ID":"8ea769ee-754a-4c3b-b34e-a04259907993","Type":"ContainerStarted","Data":"900b9f5d7ce00a9e995f1149f11a68b3db3b4c8d28045756ff12dcd986e0dce6"}
Jul 06 21:55:47.582500 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:47.582470    2550 generic.go:356] "Generic (PLEG): container finished" podID="8ea769ee-754a-4c3b-b34e-a04259907993" containerID="a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d" exitCode=0
Jul 06 21:55:47.582907 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:47.582553    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" event={"ID":"8ea769ee-754a-4c3b-b34e-a04259907993","Type":"ContainerDied","Data":"a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d"}
Jul 06 21:55:48.588523 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:48.588488    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" event={"ID":"8ea769ee-754a-4c3b-b34e-a04259907993","Type":"ContainerStarted","Data":"473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624"}
Jul 06 21:55:48.588935 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:48.588608    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:55:48.607802 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:55:48.607738    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" podStartSLOduration=2.950110931 podStartE2EDuration="3.607722589s" totalImagesPullingTime="0s" totalInitContainerRuntime="657.611658ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:55:48.606963576 +0000 UTC m=+4861.548267459" watchObservedRunningTime="2026-07-06 21:55:48.607722589 +0000 UTC m=+4861.549026436"
Jul 06 21:56:19.594753 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:56:19.594723    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:58:06.102670 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.102620    2550 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" podUID="8ea769ee-754a-4c3b-b34e-a04259907993" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 06 21:58:06.131936 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.131905    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"]
Jul 06 21:58:06.142967 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:58:06.142928    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:58:06.142967 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:58:06.142967 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:58:06.142967 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624" cmd=["/bin/sleep","15"]
Jul 06 21:58:06.143364 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.142983    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" message="" err=<
Jul 06 21:58:06.143364 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:58:06.143364 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:58:06.143364 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:58:06.143364 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:58:06.142995    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:58:06.143364 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:58:06.143364 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:58:06.143364 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" podUID="8ea769ee-754a-4c3b-b34e-a04259907993" containerName="main" containerID="cri-o://473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624"
Jul 06 21:58:06.143364 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.143053    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" podUID="8ea769ee-754a-4c3b-b34e-a04259907993" containerName="main" containerID="cri-o://473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624" gracePeriod=60
Jul 06 21:58:06.176804 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:58:06.176781    2550 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea769ee_754a_4c3b_b34e_a04259907993.slice/crio-473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624.scope\": RecentStats: unable to find data in memory cache]"
Jul 06 21:58:06.385739 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.385714    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:58:06.463555 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.463522    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-tokenizer-uds\") pod \"8ea769ee-754a-4c3b-b34e-a04259907993\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") "
Jul 06 21:58:06.463725 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.463590    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcvng\" (UniqueName: \"kubernetes.io/projected/8ea769ee-754a-4c3b-b34e-a04259907993-kube-api-access-rcvng\") pod \"8ea769ee-754a-4c3b-b34e-a04259907993\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") "
Jul 06 21:58:06.463725 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.463622    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea769ee-754a-4c3b-b34e-a04259907993-tls-certs\") pod \"8ea769ee-754a-4c3b-b34e-a04259907993\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") "
Jul 06 21:58:06.463725 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.463650    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-kserve-provision-location\") pod \"8ea769ee-754a-4c3b-b34e-a04259907993\" (UID: \"8ea769ee-754a-4c3b-b34e-a04259907993\") "
Jul 06 21:58:06.463926 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.463771    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8ea769ee-754a-4c3b-b34e-a04259907993" (UID: "8ea769ee-754a-4c3b-b34e-a04259907993"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:06.464370 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.464344    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ea769ee-754a-4c3b-b34e-a04259907993" (UID: "8ea769ee-754a-4c3b-b34e-a04259907993"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:06.465633 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.465613    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea769ee-754a-4c3b-b34e-a04259907993-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ea769ee-754a-4c3b-b34e-a04259907993" (UID: "8ea769ee-754a-4c3b-b34e-a04259907993"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:58:06.465719 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.465682    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea769ee-754a-4c3b-b34e-a04259907993-kube-api-access-rcvng" (OuterVolumeSpecName: "kube-api-access-rcvng") pod "8ea769ee-754a-4c3b-b34e-a04259907993" (UID: "8ea769ee-754a-4c3b-b34e-a04259907993"). InnerVolumeSpecName "kube-api-access-rcvng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:58:06.564746 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.564708    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:58:06.564746 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.564737    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcvng\" (UniqueName: \"kubernetes.io/projected/8ea769ee-754a-4c3b-b34e-a04259907993-kube-api-access-rcvng\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:58:06.564746 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.564746    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea769ee-754a-4c3b-b34e-a04259907993-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:58:06.564746 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:06.564754    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea769ee-754a-4c3b-b34e-a04259907993-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:58:07.202823 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.202786    2550 generic.go:356] "Generic (PLEG): container finished" podID="8ea769ee-754a-4c3b-b34e-a04259907993" containerID="473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624" exitCode=0
Jul 06 21:58:07.203228 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.202847    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" event={"ID":"8ea769ee-754a-4c3b-b34e-a04259907993","Type":"ContainerDied","Data":"473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624"}
Jul 06 21:58:07.203228 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.202862    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"
Jul 06 21:58:07.203228 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.202900    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr" event={"ID":"8ea769ee-754a-4c3b-b34e-a04259907993","Type":"ContainerDied","Data":"900b9f5d7ce00a9e995f1149f11a68b3db3b4c8d28045756ff12dcd986e0dce6"}
Jul 06 21:58:07.203228 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.202921    2550 scope.go:117] "RemoveContainer" containerID="473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624"
Jul 06 21:58:07.217762 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.217743    2550 scope.go:117] "RemoveContainer" containerID="a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d"
Jul 06 21:58:07.228722 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.228696    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"]
Jul 06 21:58:07.230013 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.229991    2550 scope.go:117] "RemoveContainer" containerID="473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624"
Jul 06 21:58:07.230484 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:58:07.230452    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624\": container with ID starting with 473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624 not found: ID does not exist" containerID="473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624"
Jul 06 21:58:07.230484 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.230490    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624"} err="failed to get container status \"473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624\": rpc error: code = NotFound desc = could not find container \"473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624\": container with ID starting with 473c259ed40cc2cd248b15fe83abec777550e90233fb0f2399310b395ad0d624 not found: ID does not exist"
Jul 06 21:58:07.230676 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.230506    2550 scope.go:117] "RemoveContainer" containerID="a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d"
Jul 06 21:58:07.230871 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:58:07.230788    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d\": container with ID starting with a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d not found: ID does not exist" containerID="a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d"
Jul 06 21:58:07.230871 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.230840    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d"} err="failed to get container status \"a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d\": rpc error: code = NotFound desc = could not find container \"a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d\": container with ID starting with a9f0771a53f81d7bfb046515d0194ebf65f8023dbc55084be9ada5344e68a12d not found: ID does not exist"
Jul 06 21:58:07.234713 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.234691    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr"]
Jul 06 21:58:07.473662 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:07.473567    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea769ee-754a-4c3b-b34e-a04259907993" path="/var/lib/kubelet/pods/8ea769ee-754a-4c3b-b34e-a04259907993/volumes"
Jul 06 21:58:12.639726 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.639693    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"]
Jul 06 21:58:12.648498 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:58:12.648470    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:58:12.648498 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:58:12.648498 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:58:12.648498 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0" cmd=["/bin/sleep","15"]
Jul 06 21:58:12.648708 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.648513    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz" message="" err=<
Jul 06 21:58:12.648708 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:58:12.648708 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:58:12.648708 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 21:58:12.648708 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:58:12.648526    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:58:12.648708 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:58:12.648708 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 21:58:12.648708 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz" podUID="0f488f52-c30a-487a-b5c0-72f333a2f8b8" containerName="main" containerID="cri-o://13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0"
Jul 06 21:58:12.648708 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.648561    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz" podUID="0f488f52-c30a-487a-b5c0-72f333a2f8b8" containerName="main" containerID="cri-o://13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0" gracePeriod=60
Jul 06 21:58:12.890448 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.890388    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:58:12.922143 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.922113    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsh5z\" (UniqueName: \"kubernetes.io/projected/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kube-api-access-wsh5z\") pod \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") "
Jul 06 21:58:12.922298 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.922166    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kserve-provision-location\") pod \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") "
Jul 06 21:58:12.922298 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.922192    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tls-certs\") pod \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") "
Jul 06 21:58:12.922298 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.922229    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tokenizer-uds\") pod \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\" (UID: \"0f488f52-c30a-487a-b5c0-72f333a2f8b8\") "
Jul 06 21:58:12.922539 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.922506    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0f488f52-c30a-487a-b5c0-72f333a2f8b8" (UID: "0f488f52-c30a-487a-b5c0-72f333a2f8b8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:12.922850 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.922823    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f488f52-c30a-487a-b5c0-72f333a2f8b8" (UID: "0f488f52-c30a-487a-b5c0-72f333a2f8b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:12.924503 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.924482    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f488f52-c30a-487a-b5c0-72f333a2f8b8" (UID: "0f488f52-c30a-487a-b5c0-72f333a2f8b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:58:12.924611 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:12.924527    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kube-api-access-wsh5z" (OuterVolumeSpecName: "kube-api-access-wsh5z") pod "0f488f52-c30a-487a-b5c0-72f333a2f8b8" (UID: "0f488f52-c30a-487a-b5c0-72f333a2f8b8"). InnerVolumeSpecName "kube-api-access-wsh5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:58:13.023280 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.023256    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsh5z\" (UniqueName: \"kubernetes.io/projected/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kube-api-access-wsh5z\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:58:13.023280 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.023278    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-kserve-provision-location\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:58:13.023427 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.023289    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:58:13.023427 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.023299    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f488f52-c30a-487a-b5c0-72f333a2f8b8-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:58:13.229911 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.229827    2550 generic.go:356] "Generic (PLEG): container finished" podID="0f488f52-c30a-487a-b5c0-72f333a2f8b8" containerID="13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0" exitCode=0
Jul 06 21:58:13.229911 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.229883    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"
Jul 06 21:58:13.229911 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.229888    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz" event={"ID":"0f488f52-c30a-487a-b5c0-72f333a2f8b8","Type":"ContainerDied","Data":"13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0"}
Jul 06 21:58:13.230125 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.229938    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz" event={"ID":"0f488f52-c30a-487a-b5c0-72f333a2f8b8","Type":"ContainerDied","Data":"a69cb444b2d008d6b157d4741578abc881a2ce778df5a633fb4d802d36f642af"}
Jul 06 21:58:13.230125 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.229965    2550 scope.go:117] "RemoveContainer" containerID="13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0"
Jul 06 21:58:13.239129 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.239113    2550 scope.go:117] "RemoveContainer" containerID="c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b"
Jul 06 21:58:13.253715 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.253689    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"]
Jul 06 21:58:13.254534 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.254514    2550 scope.go:117] "RemoveContainer" containerID="13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0"
Jul 06 21:58:13.254876 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:58:13.254853    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0\": container with ID starting with 13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0 not found: ID does not exist" containerID="13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0"
Jul 06 21:58:13.254969 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.254885    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0"} err="failed to get container status \"13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0\": rpc error: code = NotFound desc = could not find container \"13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0\": container with ID starting with 13709c7986e15a180be3273e5dd7dd1bbd3a73710459a63d66d1f103ae4dd3d0 not found: ID does not exist"
Jul 06 21:58:13.254969 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.254904    2550 scope.go:117] "RemoveContainer" containerID="c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b"
Jul 06 21:58:13.255261 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:58:13.255213    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b\": container with ID starting with c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b not found: ID does not exist" containerID="c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b"
Jul 06 21:58:13.255409 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.255271    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b"} err="failed to get container status \"c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b\": rpc error: code = NotFound desc = could not find container \"c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b\": container with ID starting with c3a1b8b9217e8751f3373e0322676422a7c64c22b80e739281393a10b034bd9b not found: ID does not exist"
Jul 06 21:58:13.257303 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.257284    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz"]
Jul 06 21:58:13.473774 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:58:13.473745    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f488f52-c30a-487a-b5c0-72f333a2f8b8" path="/var/lib/kubelet/pods/0f488f52-c30a-487a-b5c0-72f333a2f8b8/volumes"
Jul 06 21:59:01.464892 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.464858    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"]
Jul 06 21:59:01.465378 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.465302    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea769ee-754a-4c3b-b34e-a04259907993" containerName="main"
Jul 06 21:59:01.465378 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.465320    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea769ee-754a-4c3b-b34e-a04259907993" containerName="main"
Jul 06 21:59:01.465378 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.465342    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f488f52-c30a-487a-b5c0-72f333a2f8b8" containerName="storage-initializer"
Jul 06 21:59:01.465378 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.465351    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f488f52-c30a-487a-b5c0-72f333a2f8b8" containerName="storage-initializer"
Jul 06 21:59:01.465378 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.465368    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea769ee-754a-4c3b-b34e-a04259907993" containerName="storage-initializer"
Jul 06 21:59:01.465378 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.465375    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea769ee-754a-4c3b-b34e-a04259907993" containerName="storage-initializer"
Jul 06 21:59:01.465698 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.465393    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f488f52-c30a-487a-b5c0-72f333a2f8b8" containerName="main"
Jul 06 21:59:01.465698 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.465400    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f488f52-c30a-487a-b5c0-72f333a2f8b8" containerName="main"
Jul 06 21:59:01.465698 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.465487    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea769ee-754a-4c3b-b34e-a04259907993" containerName="main"
Jul 06 21:59:01.465698 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.465498    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f488f52-c30a-487a-b5c0-72f333a2f8b8" containerName="main"
Jul 06 21:59:01.470267 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.470243    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.473918 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.473890    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:59:01.473918 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.473914    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 06 21:59:01.479216 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.479190    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"]
Jul 06 21:59:01.529563 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.529532    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.529563 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.529566    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.529742 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.529582    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m4zf\" (UniqueName: \"kubernetes.io/projected/c3827074-1425-4240-94a8-cf9f27be99f2-kube-api-access-6m4zf\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.529742 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.529654    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.529742 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.529723    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.529891 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.529781    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3827074-1425-4240-94a8-cf9f27be99f2-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.630978 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.630955    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.631135 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.630992    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3827074-1425-4240-94a8-cf9f27be99f2-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.631201 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.631171    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.631201 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.631196    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.631325 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.631210    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4zf\" (UniqueName: \"kubernetes.io/projected/c3827074-1425-4240-94a8-cf9f27be99f2-kube-api-access-6m4zf\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.631325 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.631236    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.631429 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.631372    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.631598 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.631569    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.631752 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.631602    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.633274 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.633258    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.633440 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.633424    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3827074-1425-4240-94a8-cf9f27be99f2-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.639919 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.639899    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m4zf\" (UniqueName: \"kubernetes.io/projected/c3827074-1425-4240-94a8-cf9f27be99f2-kube-api-access-6m4zf\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.781647 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.781564    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:01.913753 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:01.913724    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"]
Jul 06 21:59:01.915740 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:59:01.915712    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3827074_1425_4240_94a8_cf9f27be99f2.slice/crio-561c765ec48f7ef63087a72d92ffd6b9d5d9753b4aa185344a0e2caaf7edff6a WatchSource:0}: Error finding container 561c765ec48f7ef63087a72d92ffd6b9d5d9753b4aa185344a0e2caaf7edff6a: Status 404 returned error can't find the container with id 561c765ec48f7ef63087a72d92ffd6b9d5d9753b4aa185344a0e2caaf7edff6a
Jul 06 21:59:02.450520 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:02.450487    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t" event={"ID":"c3827074-1425-4240-94a8-cf9f27be99f2","Type":"ContainerStarted","Data":"07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2"}
Jul 06 21:59:02.450693 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:02.450537    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t" event={"ID":"c3827074-1425-4240-94a8-cf9f27be99f2","Type":"ContainerStarted","Data":"561c765ec48f7ef63087a72d92ffd6b9d5d9753b4aa185344a0e2caaf7edff6a"}
Jul 06 21:59:02.479073 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:02.479026    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t" podStartSLOduration=1.479012072 podStartE2EDuration="1.479012072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:59:02.477986804 +0000 UTC m=+5055.419290652" watchObservedRunningTime="2026-07-06 21:59:02.479012072 +0000 UTC m=+5055.420315919"
Jul 06 21:59:02.782693 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:02.782612    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:11.782161 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:11.782118    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:11.800292 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:11.800265    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:12.504167 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:12.504137    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:18.761422 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.761388    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"]
Jul 06 21:59:18.764854 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.764835    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:18.768498 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.768477    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-epp-sa-dockercfg-42x9x\""
Jul 06 21:59:18.768624 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.768507    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 06 21:59:18.774014 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.773994    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"]
Jul 06 21:59:18.884584 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.884544    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:18.884835 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.884599    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvs9d\" (UniqueName: \"kubernetes.io/projected/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-kube-api-access-zvs9d\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:18.884835 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.884668    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:18.986014 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.985982    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:18.986173 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.986035    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvs9d\" (UniqueName: \"kubernetes.io/projected/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-kube-api-access-zvs9d\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:18.986173 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.986084    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:18.986506 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.986488    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:18.988385 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.988364    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:18.993665 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:18.993641    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvs9d\" (UniqueName: \"kubernetes.io/projected/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-kube-api-access-zvs9d\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:19.075653 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:19.075623    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:19.205733 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:19.205707    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"]
Jul 06 21:59:19.207546 ip-10-0-131-45 kubenswrapper[2550]: W0706 21:59:19.207518    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ab42fa_af2e_4bff_8865_86fbeb07a67c.slice/crio-45d7bb7aaf13c432951694be3ee2d4de7d0ccc7a63c47b66612f63ca68fb1861 WatchSource:0}: Error finding container 45d7bb7aaf13c432951694be3ee2d4de7d0ccc7a63c47b66612f63ca68fb1861: Status 404 returned error can't find the container with id 45d7bb7aaf13c432951694be3ee2d4de7d0ccc7a63c47b66612f63ca68fb1861
Jul 06 21:59:19.525474 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:19.525372    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r" event={"ID":"a5ab42fa-af2e-4bff-8865-86fbeb07a67c","Type":"ContainerStarted","Data":"7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500"}
Jul 06 21:59:19.525474 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:19.525424    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r" event={"ID":"a5ab42fa-af2e-4bff-8865-86fbeb07a67c","Type":"ContainerStarted","Data":"45d7bb7aaf13c432951694be3ee2d4de7d0ccc7a63c47b66612f63ca68fb1861"}
Jul 06 21:59:19.525474 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:19.525449    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:19.543528 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:19.543477    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r" podStartSLOduration=1.543463872 podStartE2EDuration="1.543463872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:59:19.542961002 +0000 UTC m=+5072.484264850" watchObservedRunningTime="2026-07-06 21:59:19.543463872 +0000 UTC m=+5072.484767719"
Jul 06 21:59:34.958981 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:34.958945    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"]
Jul 06 21:59:49.979831 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:49.979774    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t" podUID="c3827074-1425-4240-94a8-cf9f27be99f2" containerName="main" containerID="cri-o://07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2" gracePeriod=45
Jul 06 21:59:50.233450 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.233399    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:50.342532 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.342501    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-tmp-dir\") pod \"c3827074-1425-4240-94a8-cf9f27be99f2\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") "
Jul 06 21:59:50.342693 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.342555    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m4zf\" (UniqueName: \"kubernetes.io/projected/c3827074-1425-4240-94a8-cf9f27be99f2-kube-api-access-6m4zf\") pod \"c3827074-1425-4240-94a8-cf9f27be99f2\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") "
Jul 06 21:59:50.342693 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.342576    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3827074-1425-4240-94a8-cf9f27be99f2-tls-certs\") pod \"c3827074-1425-4240-94a8-cf9f27be99f2\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") "
Jul 06 21:59:50.342693 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.342616    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-dshm\") pod \"c3827074-1425-4240-94a8-cf9f27be99f2\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") "
Jul 06 21:59:50.342693 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.342660    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-home\") pod \"c3827074-1425-4240-94a8-cf9f27be99f2\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") "
Jul 06 21:59:50.342952 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.342695    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-model-cache\") pod \"c3827074-1425-4240-94a8-cf9f27be99f2\" (UID: \"c3827074-1425-4240-94a8-cf9f27be99f2\") "
Jul 06 21:59:50.342952 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.342790    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3827074-1425-4240-94a8-cf9f27be99f2" (UID: "c3827074-1425-4240-94a8-cf9f27be99f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:50.342952 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.342888    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-home" (OuterVolumeSpecName: "home") pod "c3827074-1425-4240-94a8-cf9f27be99f2" (UID: "c3827074-1425-4240-94a8-cf9f27be99f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:50.343073 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.342976    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:59:50.343073 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.343000    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:59:50.343073 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.343006    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3827074-1425-4240-94a8-cf9f27be99f2" (UID: "c3827074-1425-4240-94a8-cf9f27be99f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:50.344696 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.344674    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3827074-1425-4240-94a8-cf9f27be99f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3827074-1425-4240-94a8-cf9f27be99f2" (UID: "c3827074-1425-4240-94a8-cf9f27be99f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:59:50.344764 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.344717    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-dshm" (OuterVolumeSpecName: "dshm") pod "c3827074-1425-4240-94a8-cf9f27be99f2" (UID: "c3827074-1425-4240-94a8-cf9f27be99f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:50.344800 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.344789    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3827074-1425-4240-94a8-cf9f27be99f2-kube-api-access-6m4zf" (OuterVolumeSpecName: "kube-api-access-6m4zf") pod "c3827074-1425-4240-94a8-cf9f27be99f2" (UID: "c3827074-1425-4240-94a8-cf9f27be99f2"). InnerVolumeSpecName "kube-api-access-6m4zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:59:50.443463 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.443438    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:59:50.443463 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.443459    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3827074-1425-4240-94a8-cf9f27be99f2-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:59:50.443463 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.443467    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m4zf\" (UniqueName: \"kubernetes.io/projected/c3827074-1425-4240-94a8-cf9f27be99f2-kube-api-access-6m4zf\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:59:50.443632 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.443475    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3827074-1425-4240-94a8-cf9f27be99f2-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 21:59:50.530584 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.530525    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 21:59:50.660607 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.660572    2550 generic.go:356] "Generic (PLEG): container finished" podID="c3827074-1425-4240-94a8-cf9f27be99f2" containerID="07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2" exitCode=0
Jul 06 21:59:50.660762 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.660624    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"
Jul 06 21:59:50.660762 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.660656    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t" event={"ID":"c3827074-1425-4240-94a8-cf9f27be99f2","Type":"ContainerDied","Data":"07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2"}
Jul 06 21:59:50.660762 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.660705    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t" event={"ID":"c3827074-1425-4240-94a8-cf9f27be99f2","Type":"ContainerDied","Data":"561c765ec48f7ef63087a72d92ffd6b9d5d9753b4aa185344a0e2caaf7edff6a"}
Jul 06 21:59:50.660762 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.660726    2550 scope.go:117] "RemoveContainer" containerID="07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2"
Jul 06 21:59:50.669071 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.669056    2550 scope.go:117] "RemoveContainer" containerID="07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2"
Jul 06 21:59:50.669300 ip-10-0-131-45 kubenswrapper[2550]: E0706 21:59:50.669285    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2\": container with ID starting with 07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2 not found: ID does not exist" containerID="07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2"
Jul 06 21:59:50.669344 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.669307    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2"} err="failed to get container status \"07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2\": rpc error: code = NotFound desc = could not find container \"07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2\": container with ID starting with 07c464bdf6c59d1ac59502bc91ae730821771d2d460586beb96bc90ff3a17cd2 not found: ID does not exist"
Jul 06 21:59:50.682330 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.682306    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"]
Jul 06 21:59:50.687799 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:50.687779    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t"]
Jul 06 21:59:51.476991 ip-10-0-131-45 kubenswrapper[2550]: I0706 21:59:51.475262    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3827074-1425-4240-94a8-cf9f27be99f2" path="/var/lib/kubelet/pods/c3827074-1425-4240-94a8-cf9f27be99f2/volumes"
Jul 06 22:00:25.996417 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:25.996340    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"]
Jul 06 22:00:26.005489 ip-10-0-131-45 kubenswrapper[2550]: E0706 22:00:26.005456    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 22:00:26.005489 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:00:26.005489 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 22:00:26.005489 ip-10-0-131-45 kubenswrapper[2550]:  > containerID="7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500" cmd=["/bin/sleep","15"]
Jul 06 22:00:26.005789 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.005508    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r" message="" err=<
Jul 06 22:00:26.005789 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:00:26.005789 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 22:00:26.005789 ip-10-0-131-45 kubenswrapper[2550]:  >
Jul 06 22:00:26.005789 ip-10-0-131-45 kubenswrapper[2550]: E0706 22:00:26.005523    2550 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 22:00:26.005789 ip-10-0-131-45 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:00:26.005789 ip-10-0-131-45 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Jul 06 22:00:26.005789 ip-10-0-131-45 kubenswrapper[2550]:  > pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r" podUID="a5ab42fa-af2e-4bff-8865-86fbeb07a67c" containerName="main" containerID="cri-o://7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500"
Jul 06 22:00:26.005789 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.005587    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r" podUID="a5ab42fa-af2e-4bff-8865-86fbeb07a67c" containerName="main" containerID="cri-o://7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500" gracePeriod=60
Jul 06 22:00:26.259002 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.258944    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 22:00:26.354927 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.354895    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tokenizer-uds\") pod \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") "
Jul 06 22:00:26.355101 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.354986    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvs9d\" (UniqueName: \"kubernetes.io/projected/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-kube-api-access-zvs9d\") pod \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") "
Jul 06 22:00:26.355101 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.355008    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tls-certs\") pod \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\" (UID: \"a5ab42fa-af2e-4bff-8865-86fbeb07a67c\") "
Jul 06 22:00:26.355218 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.355188    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a5ab42fa-af2e-4bff-8865-86fbeb07a67c" (UID: "a5ab42fa-af2e-4bff-8865-86fbeb07a67c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:00:26.357003 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.356975    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5ab42fa-af2e-4bff-8865-86fbeb07a67c" (UID: "a5ab42fa-af2e-4bff-8865-86fbeb07a67c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:00:26.357112 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.357058    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-kube-api-access-zvs9d" (OuterVolumeSpecName: "kube-api-access-zvs9d") pod "a5ab42fa-af2e-4bff-8865-86fbeb07a67c" (UID: "a5ab42fa-af2e-4bff-8865-86fbeb07a67c"). InnerVolumeSpecName "kube-api-access-zvs9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:00:26.456069 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.456038    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tokenizer-uds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:00:26.456069 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.456064    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvs9d\" (UniqueName: \"kubernetes.io/projected/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-kube-api-access-zvs9d\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:00:26.456069 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.456073    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ab42fa-af2e-4bff-8865-86fbeb07a67c-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:00:26.817580 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.817539    2550 generic.go:356] "Generic (PLEG): container finished" podID="a5ab42fa-af2e-4bff-8865-86fbeb07a67c" containerID="7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500" exitCode=0
Jul 06 22:00:26.817752 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.817600    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r" event={"ID":"a5ab42fa-af2e-4bff-8865-86fbeb07a67c","Type":"ContainerDied","Data":"7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500"}
Jul 06 22:00:26.818305 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.818009    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r" event={"ID":"a5ab42fa-af2e-4bff-8865-86fbeb07a67c","Type":"ContainerDied","Data":"45d7bb7aaf13c432951694be3ee2d4de7d0ccc7a63c47b66612f63ca68fb1861"}
Jul 06 22:00:26.818305 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.818043    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"
Jul 06 22:00:26.818305 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.818079    2550 scope.go:117] "RemoveContainer" containerID="7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500"
Jul 06 22:00:26.832587 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.832571    2550 scope.go:117] "RemoveContainer" containerID="7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500"
Jul 06 22:00:26.832864 ip-10-0-131-45 kubenswrapper[2550]: E0706 22:00:26.832841    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500\": container with ID starting with 7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500 not found: ID does not exist" containerID="7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500"
Jul 06 22:00:26.832960 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.832869    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500"} err="failed to get container status \"7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500\": rpc error: code = NotFound desc = could not find container \"7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500\": container with ID starting with 7f1ae9bf50c22b0e33cb31c784188c47045ce0d469e214df23fc56047b20d500 not found: ID does not exist"
Jul 06 22:00:26.842748 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.842724    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"]
Jul 06 22:00:26.849057 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:26.849034    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r"]
Jul 06 22:00:27.474030 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:27.473995    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ab42fa-af2e-4bff-8865-86fbeb07a67c" path="/var/lib/kubelet/pods/a5ab42fa-af2e-4bff-8865-86fbeb07a67c/volumes"
Jul 06 22:00:47.423648 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.423617    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-555fd59bd5-82brg"]
Jul 06 22:00:47.424038 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.423961    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3827074-1425-4240-94a8-cf9f27be99f2" containerName="main"
Jul 06 22:00:47.424038 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.423972    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3827074-1425-4240-94a8-cf9f27be99f2" containerName="main"
Jul 06 22:00:47.424038 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.423983    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ab42fa-af2e-4bff-8865-86fbeb07a67c" containerName="main"
Jul 06 22:00:47.424038 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.423989    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ab42fa-af2e-4bff-8865-86fbeb07a67c" containerName="main"
Jul 06 22:00:47.424166 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.424070    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ab42fa-af2e-4bff-8865-86fbeb07a67c" containerName="main"
Jul 06 22:00:47.424166 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.424079    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3827074-1425-4240-94a8-cf9f27be99f2" containerName="main"
Jul 06 22:00:47.427001 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.426985    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg"
Jul 06 22:00:47.430858 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.430836    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 06 22:00:47.430992 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.430947    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-xnw54\""
Jul 06 22:00:47.430992 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.430983    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 06 22:00:47.431112 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.431070    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 06 22:00:47.437400 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.437381    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-555fd59bd5-82brg"]
Jul 06 22:00:47.524322 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.524292    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwf89\" (UniqueName: \"kubernetes.io/projected/9b9f8294-9be8-4620-a54f-c1f048188734-kube-api-access-lwf89\") pod \"llmisvc-controller-manager-555fd59bd5-82brg\" (UID: \"9b9f8294-9be8-4620-a54f-c1f048188734\") " pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg"
Jul 06 22:00:47.524480 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.524372    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b9f8294-9be8-4620-a54f-c1f048188734-cert\") pod \"llmisvc-controller-manager-555fd59bd5-82brg\" (UID: \"9b9f8294-9be8-4620-a54f-c1f048188734\") " pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg"
Jul 06 22:00:47.625648 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.625602    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwf89\" (UniqueName: \"kubernetes.io/projected/9b9f8294-9be8-4620-a54f-c1f048188734-kube-api-access-lwf89\") pod \"llmisvc-controller-manager-555fd59bd5-82brg\" (UID: \"9b9f8294-9be8-4620-a54f-c1f048188734\") " pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg"
Jul 06 22:00:47.625863 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.625696    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b9f8294-9be8-4620-a54f-c1f048188734-cert\") pod \"llmisvc-controller-manager-555fd59bd5-82brg\" (UID: \"9b9f8294-9be8-4620-a54f-c1f048188734\") " pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg"
Jul 06 22:00:47.628019 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.627997    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b9f8294-9be8-4620-a54f-c1f048188734-cert\") pod \"llmisvc-controller-manager-555fd59bd5-82brg\" (UID: \"9b9f8294-9be8-4620-a54f-c1f048188734\") " pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg"
Jul 06 22:00:47.634938 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.634918    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwf89\" (UniqueName: \"kubernetes.io/projected/9b9f8294-9be8-4620-a54f-c1f048188734-kube-api-access-lwf89\") pod \"llmisvc-controller-manager-555fd59bd5-82brg\" (UID: \"9b9f8294-9be8-4620-a54f-c1f048188734\") " pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg"
Jul 06 22:00:47.737986 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.737910    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg"
Jul 06 22:00:47.864834 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.864795    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-555fd59bd5-82brg"]
Jul 06 22:00:47.866833 ip-10-0-131-45 kubenswrapper[2550]: W0706 22:00:47.866794    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9b9f8294_9be8_4620_a54f_c1f048188734.slice/crio-c126af0275cd30e97efef323dfddad35c83315a87b346b771b64bc58aeb6e343 WatchSource:0}: Error finding container c126af0275cd30e97efef323dfddad35c83315a87b346b771b64bc58aeb6e343: Status 404 returned error can't find the container with id c126af0275cd30e97efef323dfddad35c83315a87b346b771b64bc58aeb6e343
Jul 06 22:00:47.868116 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.868100    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 22:00:47.911299 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:47.911269    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg" event={"ID":"9b9f8294-9be8-4620-a54f-c1f048188734","Type":"ContainerStarted","Data":"c126af0275cd30e97efef323dfddad35c83315a87b346b771b64bc58aeb6e343"}
Jul 06 22:00:51.930293 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:51.930253    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg" event={"ID":"9b9f8294-9be8-4620-a54f-c1f048188734","Type":"ContainerStarted","Data":"81921e172850c5dc67afdc4bf1343ef197f480de8bee97c0caa74ddbb39b9bf6"}
Jul 06 22:00:51.930666 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:51.930381    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg"
Jul 06 22:00:51.951542 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:00:51.951493    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg" podStartSLOduration=1.596101346 podStartE2EDuration="4.951478849s" totalImagesPullingTime="3.355377503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:00:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:00:51.950074664 +0000 UTC m=+5164.891378513" watchObservedRunningTime="2026-07-06 22:00:51.951478849 +0000 UTC m=+5164.892782724"
Jul 06 22:01:19.363948 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.363918    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"]
Jul 06 22:01:19.367224 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.367209    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.371047 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.371029    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 06 22:01:19.371154 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.371068    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 22:01:19.376975 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.376953    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"]
Jul 06 22:01:19.406121 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.406094    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.406247 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.406139    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.406247 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.406169    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.406247 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.406212    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzwpp\" (UniqueName: \"kubernetes.io/projected/90443779-1933-4342-9502-53d47485167e-kube-api-access-rzwpp\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.406390 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.406268    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.406390 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.406308    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90443779-1933-4342-9502-53d47485167e-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.507559 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.507530    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.507708 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.507582    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.507801 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.507773    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.507934 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.507915    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzwpp\" (UniqueName: \"kubernetes.io/projected/90443779-1933-4342-9502-53d47485167e-kube-api-access-rzwpp\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.508029 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.508011    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.508346 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.508319    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.508637 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.508612    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.508755 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.508656    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90443779-1933-4342-9502-53d47485167e-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.508917 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.508896    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.510033 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.510016    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.511029 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.511009    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90443779-1933-4342-9502-53d47485167e-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.517464 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.517443    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzwpp\" (UniqueName: \"kubernetes.io/projected/90443779-1933-4342-9502-53d47485167e-kube-api-access-rzwpp\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h\" (UID: \"90443779-1933-4342-9502-53d47485167e\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.678263 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.678189    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:19.828628 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:19.828585    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"]
Jul 06 22:01:19.829947 ip-10-0-131-45 kubenswrapper[2550]: W0706 22:01:19.829919    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90443779_1933_4342_9502_53d47485167e.slice/crio-07af88f0dcedec7b4ff227f337c51ad95de7f62661b4bef9e3cae4e26f69bed6 WatchSource:0}: Error finding container 07af88f0dcedec7b4ff227f337c51ad95de7f62661b4bef9e3cae4e26f69bed6: Status 404 returned error can't find the container with id 07af88f0dcedec7b4ff227f337c51ad95de7f62661b4bef9e3cae4e26f69bed6
Jul 06 22:01:20.052208 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:20.052108    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h" event={"ID":"90443779-1933-4342-9502-53d47485167e","Type":"ContainerStarted","Data":"560aa0fe3a9da9e4187f64e3a29494a22ec52c950292f9b7cbc859b7d4e22a58"}
Jul 06 22:01:20.052208 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:20.052160    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h" event={"ID":"90443779-1933-4342-9502-53d47485167e","Type":"ContainerStarted","Data":"07af88f0dcedec7b4ff227f337c51ad95de7f62661b4bef9e3cae4e26f69bed6"}
Jul 06 22:01:20.070988 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:20.070944    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h" podStartSLOduration=1.070929211 podStartE2EDuration="1.070929211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:01:20.07031998 +0000 UTC m=+5193.011623841" watchObservedRunningTime="2026-07-06 22:01:20.070929211 +0000 UTC m=+5193.012233058"
Jul 06 22:01:20.679215 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:20.679183    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:22.937884 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:22.937855    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-555fd59bd5-82brg"
Jul 06 22:01:29.678572 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:29.678536    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:29.692795 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:29.692769    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:01:30.107721 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:01:30.107695    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:02:10.104337 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:10.104309    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"]
Jul 06 22:02:25.124740 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.124681    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h" podUID="90443779-1933-4342-9502-53d47485167e" containerName="main" containerID="cri-o://560aa0fe3a9da9e4187f64e3a29494a22ec52c950292f9b7cbc859b7d4e22a58" gracePeriod=45
Jul 06 22:02:25.337948 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.337918    2550 generic.go:356] "Generic (PLEG): container finished" podID="90443779-1933-4342-9502-53d47485167e" containerID="560aa0fe3a9da9e4187f64e3a29494a22ec52c950292f9b7cbc859b7d4e22a58" exitCode=0
Jul 06 22:02:25.338089 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.337990    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h" event={"ID":"90443779-1933-4342-9502-53d47485167e","Type":"ContainerDied","Data":"560aa0fe3a9da9e4187f64e3a29494a22ec52c950292f9b7cbc859b7d4e22a58"}
Jul 06 22:02:25.375031 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.374975    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:02:25.477601 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.477574    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-tmp-dir\") pod \"90443779-1933-4342-9502-53d47485167e\" (UID: \"90443779-1933-4342-9502-53d47485167e\") "
Jul 06 22:02:25.477601 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.477601    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90443779-1933-4342-9502-53d47485167e-tls-certs\") pod \"90443779-1933-4342-9502-53d47485167e\" (UID: \"90443779-1933-4342-9502-53d47485167e\") "
Jul 06 22:02:25.477790 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.477638    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-dshm\") pod \"90443779-1933-4342-9502-53d47485167e\" (UID: \"90443779-1933-4342-9502-53d47485167e\") "
Jul 06 22:02:25.477790 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.477664    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzwpp\" (UniqueName: \"kubernetes.io/projected/90443779-1933-4342-9502-53d47485167e-kube-api-access-rzwpp\") pod \"90443779-1933-4342-9502-53d47485167e\" (UID: \"90443779-1933-4342-9502-53d47485167e\") "
Jul 06 22:02:25.477790 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.477700    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-model-cache\") pod \"90443779-1933-4342-9502-53d47485167e\" (UID: \"90443779-1933-4342-9502-53d47485167e\") "
Jul 06 22:02:25.477790 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.477748    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-home\") pod \"90443779-1933-4342-9502-53d47485167e\" (UID: \"90443779-1933-4342-9502-53d47485167e\") "
Jul 06 22:02:25.478036 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.477836    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90443779-1933-4342-9502-53d47485167e" (UID: "90443779-1933-4342-9502-53d47485167e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:02:25.478094 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.478037    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:02:25.478094 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.478046    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-model-cache" (OuterVolumeSpecName: "model-cache") pod "90443779-1933-4342-9502-53d47485167e" (UID: "90443779-1933-4342-9502-53d47485167e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:02:25.478094 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.478063    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-home" (OuterVolumeSpecName: "home") pod "90443779-1933-4342-9502-53d47485167e" (UID: "90443779-1933-4342-9502-53d47485167e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:02:25.479701 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.479681    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90443779-1933-4342-9502-53d47485167e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90443779-1933-4342-9502-53d47485167e" (UID: "90443779-1933-4342-9502-53d47485167e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:02:25.479768 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.479723    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90443779-1933-4342-9502-53d47485167e-kube-api-access-rzwpp" (OuterVolumeSpecName: "kube-api-access-rzwpp") pod "90443779-1933-4342-9502-53d47485167e" (UID: "90443779-1933-4342-9502-53d47485167e"). InnerVolumeSpecName "kube-api-access-rzwpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:02:25.479768 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.479763    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-dshm" (OuterVolumeSpecName: "dshm") pod "90443779-1933-4342-9502-53d47485167e" (UID: "90443779-1933-4342-9502-53d47485167e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:02:25.579349 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.579328    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:02:25.579349 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.579350    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90443779-1933-4342-9502-53d47485167e-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:02:25.579565 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.579364    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:02:25.579565 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.579376    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzwpp\" (UniqueName: \"kubernetes.io/projected/90443779-1933-4342-9502-53d47485167e-kube-api-access-rzwpp\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:02:25.579565 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:25.579387    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90443779-1933-4342-9502-53d47485167e-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:02:26.343434 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:26.343412    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"
Jul 06 22:02:26.343860 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:26.343404    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h" event={"ID":"90443779-1933-4342-9502-53d47485167e","Type":"ContainerDied","Data":"07af88f0dcedec7b4ff227f337c51ad95de7f62661b4bef9e3cae4e26f69bed6"}
Jul 06 22:02:26.343860 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:26.343541    2550 scope.go:117] "RemoveContainer" containerID="560aa0fe3a9da9e4187f64e3a29494a22ec52c950292f9b7cbc859b7d4e22a58"
Jul 06 22:02:26.376367 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:26.376339    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"]
Jul 06 22:02:26.381005 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:26.380986    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h"]
Jul 06 22:02:27.474356 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:27.474325    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90443779-1933-4342-9502-53d47485167e" path="/var/lib/kubelet/pods/90443779-1933-4342-9502-53d47485167e/volumes"
Jul 06 22:02:49.972905 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:49.972870    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"]
Jul 06 22:02:49.973399 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:49.973369    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90443779-1933-4342-9502-53d47485167e" containerName="main"
Jul 06 22:02:49.973399 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:49.973388    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="90443779-1933-4342-9502-53d47485167e" containerName="main"
Jul 06 22:02:49.973515 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:49.973470    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="90443779-1933-4342-9502-53d47485167e" containerName="main"
Jul 06 22:02:49.977641 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:49.977620    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:49.981468 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:49.981429    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 22:02:49.981617 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:49.981490    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 06 22:02:49.990114 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:49.990094    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"]
Jul 06 22:02:50.096899 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.096864    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d181a552-173a-4976-a3db-4b9b1f72785b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.096899 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.096901    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.097123 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.096948    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.097123 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.096964    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.097123 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.096984    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4tt\" (UniqueName: \"kubernetes.io/projected/d181a552-173a-4976-a3db-4b9b1f72785b-kube-api-access-9l4tt\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.097123 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.097075    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.197915 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.197881    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d181a552-173a-4976-a3db-4b9b1f72785b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.198087 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.197925    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.198087 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.197985    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.198087 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.198009    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.198087 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.198037    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4tt\" (UniqueName: \"kubernetes.io/projected/d181a552-173a-4976-a3db-4b9b1f72785b-kube-api-access-9l4tt\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.198087 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.198082    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.198395 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.198369    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.198486 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.198425    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.198546 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.198507    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.200095 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.200075    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.200504 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.200483    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d181a552-173a-4976-a3db-4b9b1f72785b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.209120 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.209101    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4tt\" (UniqueName: \"kubernetes.io/projected/d181a552-173a-4976-a3db-4b9b1f72785b-kube-api-access-9l4tt\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.287981 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.287914    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:02:50.422506 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.422483    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"]
Jul 06 22:02:50.424463 ip-10-0-131-45 kubenswrapper[2550]: W0706 22:02:50.424440    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd181a552_173a_4976_a3db_4b9b1f72785b.slice/crio-fec41eca5c30c6677fdec9aacf2643f57982ab829f3be1879deeaf329b2bede5 WatchSource:0}: Error finding container fec41eca5c30c6677fdec9aacf2643f57982ab829f3be1879deeaf329b2bede5: Status 404 returned error can't find the container with id fec41eca5c30c6677fdec9aacf2643f57982ab829f3be1879deeaf329b2bede5
Jul 06 22:02:50.456724 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:50.456702    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg" event={"ID":"d181a552-173a-4976-a3db-4b9b1f72785b","Type":"ContainerStarted","Data":"fec41eca5c30c6677fdec9aacf2643f57982ab829f3be1879deeaf329b2bede5"}
Jul 06 22:02:51.461971 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:51.461934    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg" event={"ID":"d181a552-173a-4976-a3db-4b9b1f72785b","Type":"ContainerStarted","Data":"3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff"}
Jul 06 22:02:51.485408 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:51.485364    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg" podStartSLOduration=2.485349315 podStartE2EDuration="2.485349315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:02:51.482160772 +0000 UTC m=+5284.423464620" watchObservedRunningTime="2026-07-06 22:02:51.485349315 +0000 UTC m=+5284.426653162"
Jul 06 22:02:52.288337 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:02:52.288304    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:03:00.288976 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:00.288937    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:03:00.301468 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:00.301446    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:03:00.511185 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:00.511159    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:03:23.799548 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:23.799470    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"]
Jul 06 22:03:38.818868 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:38.818825    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg" podUID="d181a552-173a-4976-a3db-4b9b1f72785b" containerName="main" containerID="cri-o://3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff" gracePeriod=45
Jul 06 22:03:39.059672 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.059648    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:03:39.206568 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.206481    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d181a552-173a-4976-a3db-4b9b1f72785b-tls-certs\") pod \"d181a552-173a-4976-a3db-4b9b1f72785b\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") "
Jul 06 22:03:39.206568 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.206516    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l4tt\" (UniqueName: \"kubernetes.io/projected/d181a552-173a-4976-a3db-4b9b1f72785b-kube-api-access-9l4tt\") pod \"d181a552-173a-4976-a3db-4b9b1f72785b\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") "
Jul 06 22:03:39.206568 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.206562    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-model-cache\") pod \"d181a552-173a-4976-a3db-4b9b1f72785b\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") "
Jul 06 22:03:39.206898 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.206585    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-tmp-dir\") pod \"d181a552-173a-4976-a3db-4b9b1f72785b\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") "
Jul 06 22:03:39.206898 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.206618    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-dshm\") pod \"d181a552-173a-4976-a3db-4b9b1f72785b\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") "
Jul 06 22:03:39.206898 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.206636    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-home\") pod \"d181a552-173a-4976-a3db-4b9b1f72785b\" (UID: \"d181a552-173a-4976-a3db-4b9b1f72785b\") "
Jul 06 22:03:39.206898 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.206886    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d181a552-173a-4976-a3db-4b9b1f72785b" (UID: "d181a552-173a-4976-a3db-4b9b1f72785b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:39.207054 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.206916    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d181a552-173a-4976-a3db-4b9b1f72785b" (UID: "d181a552-173a-4976-a3db-4b9b1f72785b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:39.207054 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.206968    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-home" (OuterVolumeSpecName: "home") pod "d181a552-173a-4976-a3db-4b9b1f72785b" (UID: "d181a552-173a-4976-a3db-4b9b1f72785b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:39.208721 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.208694    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-dshm" (OuterVolumeSpecName: "dshm") pod "d181a552-173a-4976-a3db-4b9b1f72785b" (UID: "d181a552-173a-4976-a3db-4b9b1f72785b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:39.208721 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.208700    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d181a552-173a-4976-a3db-4b9b1f72785b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d181a552-173a-4976-a3db-4b9b1f72785b" (UID: "d181a552-173a-4976-a3db-4b9b1f72785b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:03:39.208888 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.208741    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d181a552-173a-4976-a3db-4b9b1f72785b-kube-api-access-9l4tt" (OuterVolumeSpecName: "kube-api-access-9l4tt") pod "d181a552-173a-4976-a3db-4b9b1f72785b" (UID: "d181a552-173a-4976-a3db-4b9b1f72785b"). InnerVolumeSpecName "kube-api-access-9l4tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:03:39.308168 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.308138    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:03:39.308168 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.308163    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:03:39.308168 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.308171    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:03:39.308388 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.308179    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d181a552-173a-4976-a3db-4b9b1f72785b-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:03:39.308388 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.308186    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d181a552-173a-4976-a3db-4b9b1f72785b-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:03:39.308388 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.308194    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l4tt\" (UniqueName: \"kubernetes.io/projected/d181a552-173a-4976-a3db-4b9b1f72785b-kube-api-access-9l4tt\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:03:39.670897 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.670858    2550 generic.go:356] "Generic (PLEG): container finished" podID="d181a552-173a-4976-a3db-4b9b1f72785b" containerID="3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff" exitCode=0
Jul 06 22:03:39.671049 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.670915    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"
Jul 06 22:03:39.671049 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.670927    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg" event={"ID":"d181a552-173a-4976-a3db-4b9b1f72785b","Type":"ContainerDied","Data":"3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff"}
Jul 06 22:03:39.671049 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.670987    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg" event={"ID":"d181a552-173a-4976-a3db-4b9b1f72785b","Type":"ContainerDied","Data":"fec41eca5c30c6677fdec9aacf2643f57982ab829f3be1879deeaf329b2bede5"}
Jul 06 22:03:39.671049 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.671017    2550 scope.go:117] "RemoveContainer" containerID="3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff"
Jul 06 22:03:39.679109 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.679093    2550 scope.go:117] "RemoveContainer" containerID="3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff"
Jul 06 22:03:39.679350 ip-10-0-131-45 kubenswrapper[2550]: E0706 22:03:39.679330    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff\": container with ID starting with 3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff not found: ID does not exist" containerID="3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff"
Jul 06 22:03:39.679416 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.679360    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff"} err="failed to get container status \"3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff\": rpc error: code = NotFound desc = could not find container \"3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff\": container with ID starting with 3a6faf6388718fe922a6bb926f5a8f76ac898dabc85508fd197a54dfed0deaff not found: ID does not exist"
Jul 06 22:03:39.688943 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.688920    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"]
Jul 06 22:03:39.692642 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:39.692621    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg"]
Jul 06 22:03:41.474658 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:41.474629    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d181a552-173a-4976-a3db-4b9b1f72785b" path="/var/lib/kubelet/pods/d181a552-173a-4976-a3db-4b9b1f72785b/volumes"
Jul 06 22:03:49.029559 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.029524    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"]
Jul 06 22:03:49.029998 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.029981    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d181a552-173a-4976-a3db-4b9b1f72785b" containerName="main"
Jul 06 22:03:49.030050 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.029999    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d181a552-173a-4976-a3db-4b9b1f72785b" containerName="main"
Jul 06 22:03:49.030088 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.030072    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d181a552-173a-4976-a3db-4b9b1f72785b" containerName="main"
Jul 06 22:03:49.034225 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.034204    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.038090 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.038072    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 22:03:49.038213 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.038076    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 06 22:03:49.050400 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.050380    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"]
Jul 06 22:03:49.197693 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.197656    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.197908 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.197722    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad17c5df-6533-4378-b417-259efe93ba3c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.197908 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.197786    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.197908 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.197866    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.197908 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.197891    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.198049 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.197959    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6lv\" (UniqueName: \"kubernetes.io/projected/ad17c5df-6533-4378-b417-259efe93ba3c-kube-api-access-vc6lv\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.299409 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.299328    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad17c5df-6533-4378-b417-259efe93ba3c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.299409 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.299366    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.299409 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.299397    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.299657 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.299419    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.299657 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.299488    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6lv\" (UniqueName: \"kubernetes.io/projected/ad17c5df-6533-4378-b417-259efe93ba3c-kube-api-access-vc6lv\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.299657 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.299521    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.299865 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.299799    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.299956 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.299931    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.300000 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.299954    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.301707 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.301675    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad17c5df-6533-4378-b417-259efe93ba3c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.301978 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.301961    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad17c5df-6533-4378-b417-259efe93ba3c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.308440 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.308420    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6lv\" (UniqueName: \"kubernetes.io/projected/ad17c5df-6533-4378-b417-259efe93ba3c-kube-api-access-vc6lv\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh\" (UID: \"ad17c5df-6533-4378-b417-259efe93ba3c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.344443 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.344419    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:49.489930 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.489904    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"]
Jul 06 22:03:49.492294 ip-10-0-131-45 kubenswrapper[2550]: W0706 22:03:49.492265    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad17c5df_6533_4378_b417_259efe93ba3c.slice/crio-6ba00c22893b777a103c6ae8c3f83c343c3ae495f33019a72fb01a01e1820867 WatchSource:0}: Error finding container 6ba00c22893b777a103c6ae8c3f83c343c3ae495f33019a72fb01a01e1820867: Status 404 returned error can't find the container with id 6ba00c22893b777a103c6ae8c3f83c343c3ae495f33019a72fb01a01e1820867
Jul 06 22:03:49.717533 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.717498    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh" event={"ID":"ad17c5df-6533-4378-b417-259efe93ba3c","Type":"ContainerStarted","Data":"74be01163ec95fc2b85536f6be3d0a1ce8269b7960270e545e7b07cc09be017f"}
Jul 06 22:03:49.717700 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.717557    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh" event={"ID":"ad17c5df-6533-4378-b417-259efe93ba3c","Type":"ContainerStarted","Data":"6ba00c22893b777a103c6ae8c3f83c343c3ae495f33019a72fb01a01e1820867"}
Jul 06 22:03:49.739554 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:49.739501    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh" podStartSLOduration=0.739486814 podStartE2EDuration="739.486814ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:03:49.739035889 +0000 UTC m=+5342.680339761" watchObservedRunningTime="2026-07-06 22:03:49.739486814 +0000 UTC m=+5342.680790688"
Jul 06 22:03:50.344911 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:50.344869    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:55.893914 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:55.893885    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"]
Jul 06 22:03:55.901782 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:55.901752    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:55.904504 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:55.904478    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 06 22:03:55.910266 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:55.909738    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"]
Jul 06 22:03:55.960209 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:55.960169    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:55.960395 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:55.960260    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:55.960395 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:55.960289    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:55.960395 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:55.960346    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:55.960395 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:55.960370    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2m48\" (UniqueName: \"kubernetes.io/projected/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-kube-api-access-t2m48\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:55.960554 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:55.960442    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.061540 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.061510    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.061540 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.061542    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2m48\" (UniqueName: \"kubernetes.io/projected/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-kube-api-access-t2m48\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.061763 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.061568    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.061763 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.061598    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.061763 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.061646    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.061763 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.061669    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.062048 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.062023    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.062114 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.062078    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.062114 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.062106    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.064080 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.064048    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.064187 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.064147    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.070669 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.070631    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2m48\" (UniqueName: \"kubernetes.io/projected/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-kube-api-access-t2m48\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.214403 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.214322    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:56.349764 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.349734    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"]
Jul 06 22:03:56.353110 ip-10-0-131-45 kubenswrapper[2550]: W0706 22:03:56.353081    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e62cd4d_81a3_4e0c_b892_9dbc91563a19.slice/crio-c4215e1b73206127c657040d6d9d172c3bd12edaca0e590c90da7f6fb2dfa55b WatchSource:0}: Error finding container c4215e1b73206127c657040d6d9d172c3bd12edaca0e590c90da7f6fb2dfa55b: Status 404 returned error can't find the container with id c4215e1b73206127c657040d6d9d172c3bd12edaca0e590c90da7f6fb2dfa55b
Jul 06 22:03:56.749049 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.749018    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx" event={"ID":"0e62cd4d-81a3-4e0c-b892-9dbc91563a19","Type":"ContainerStarted","Data":"eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f"}
Jul 06 22:03:56.749215 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.749068    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx" event={"ID":"0e62cd4d-81a3-4e0c-b892-9dbc91563a19","Type":"ContainerStarted","Data":"c4215e1b73206127c657040d6d9d172c3bd12edaca0e590c90da7f6fb2dfa55b"}
Jul 06 22:03:56.771284 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:56.771237    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx" podStartSLOduration=1.7712220570000001 podStartE2EDuration="1.771222057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:03:56.769788457 +0000 UTC m=+5349.711092306" watchObservedRunningTime="2026-07-06 22:03:56.771222057 +0000 UTC m=+5349.712525903"
Jul 06 22:03:57.214480 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:57.214447    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:03:59.345641 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:59.345596    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:59.357664 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:59.357641    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:03:59.773578 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:03:59.773498    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh"
Jul 06 22:04:06.214577 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:06.214542    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:04:06.226968 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:06.226943    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:04:06.807347 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:06.807318    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:04:29.854655 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:29.854619    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"]
Jul 06 22:04:44.874647 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:44.874562    2550 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx" podUID="0e62cd4d-81a3-4e0c-b892-9dbc91563a19" containerName="main" containerID="cri-o://eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f" gracePeriod=45
Jul 06 22:04:45.132014 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.131958    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:04:45.192137 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192116    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-dshm\") pod \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") "
Jul 06 22:04:45.192271 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192150    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2m48\" (UniqueName: \"kubernetes.io/projected/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-kube-api-access-t2m48\") pod \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") "
Jul 06 22:04:45.192271 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192202    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-home\") pod \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") "
Jul 06 22:04:45.192271 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192251    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tmp-dir\") pod \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") "
Jul 06 22:04:45.192434 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192416    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-model-cache\") pod \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") "
Jul 06 22:04:45.192549 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192450    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tls-certs\") pod \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\" (UID: \"0e62cd4d-81a3-4e0c-b892-9dbc91563a19\") "
Jul 06 22:04:45.192549 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192478    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e62cd4d-81a3-4e0c-b892-9dbc91563a19" (UID: "0e62cd4d-81a3-4e0c-b892-9dbc91563a19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:04:45.192549 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192507    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-home" (OuterVolumeSpecName: "home") pod "0e62cd4d-81a3-4e0c-b892-9dbc91563a19" (UID: "0e62cd4d-81a3-4e0c-b892-9dbc91563a19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:04:45.192760 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192705    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e62cd4d-81a3-4e0c-b892-9dbc91563a19" (UID: "0e62cd4d-81a3-4e0c-b892-9dbc91563a19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:04:45.192888 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192873    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-model-cache\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:04:45.192888 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192895    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-home\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:04:45.193022 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.192909    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tmp-dir\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:04:45.194190 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.194164    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-dshm" (OuterVolumeSpecName: "dshm") pod "0e62cd4d-81a3-4e0c-b892-9dbc91563a19" (UID: "0e62cd4d-81a3-4e0c-b892-9dbc91563a19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:04:45.194566 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.194536    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-kube-api-access-t2m48" (OuterVolumeSpecName: "kube-api-access-t2m48") pod "0e62cd4d-81a3-4e0c-b892-9dbc91563a19" (UID: "0e62cd4d-81a3-4e0c-b892-9dbc91563a19"). InnerVolumeSpecName "kube-api-access-t2m48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:04:45.194657 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.194566    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e62cd4d-81a3-4e0c-b892-9dbc91563a19" (UID: "0e62cd4d-81a3-4e0c-b892-9dbc91563a19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:04:45.293493 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.293465    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-tls-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:04:45.293493 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.293490    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-dshm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:04:45.293493 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.293499    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2m48\" (UniqueName: \"kubernetes.io/projected/0e62cd4d-81a3-4e0c-b892-9dbc91563a19-kube-api-access-t2m48\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\""
Jul 06 22:04:45.969775 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.969744    2550 generic.go:356] "Generic (PLEG): container finished" podID="0e62cd4d-81a3-4e0c-b892-9dbc91563a19" containerID="eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f" exitCode=0
Jul 06 22:04:45.970228 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.969790    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"
Jul 06 22:04:45.970228 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.969789    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx" event={"ID":"0e62cd4d-81a3-4e0c-b892-9dbc91563a19","Type":"ContainerDied","Data":"eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f"}
Jul 06 22:04:45.970228 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.969858    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx" event={"ID":"0e62cd4d-81a3-4e0c-b892-9dbc91563a19","Type":"ContainerDied","Data":"c4215e1b73206127c657040d6d9d172c3bd12edaca0e590c90da7f6fb2dfa55b"}
Jul 06 22:04:45.970228 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.969878    2550 scope.go:117] "RemoveContainer" containerID="eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f"
Jul 06 22:04:45.978137 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.978120    2550 scope.go:117] "RemoveContainer" containerID="eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f"
Jul 06 22:04:45.978397 ip-10-0-131-45 kubenswrapper[2550]: E0706 22:04:45.978379    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f\": container with ID starting with eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f not found: ID does not exist" containerID="eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f"
Jul 06 22:04:45.978443 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.978407    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f"} err="failed to get container status \"eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f\": rpc error: code = NotFound desc = could not find container \"eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f\": container with ID starting with eed8c42bf4114d7acee5551566871a21f63d70caa4f941d8776e191cbacd739f not found: ID does not exist"
Jul 06 22:04:45.990125 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.990094    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"]
Jul 06 22:04:45.994254 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:45.994234    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx"]
Jul 06 22:04:47.475285 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:04:47.475254    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e62cd4d-81a3-4e0c-b892-9dbc91563a19" path="/var/lib/kubelet/pods/0e62cd4d-81a3-4e0c-b892-9dbc91563a19/volumes"
Jul 06 22:22:46.468899 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:22:46.468869    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:05.446170 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:05.446139    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:05.519492 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:05.519466    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:07.031419 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:07.031392    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:07.073456 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:07.073428    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:08.567720 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:08.567696    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:08.608527 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:08.608501    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:10.079146 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:10.079117    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:10.125165 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:10.125139    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:11.680657 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:11.680623    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:11.728160 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:11.728133    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:13.322671 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:13.322641    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:13.371489 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:13.371469    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:14.902718 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:14.902690    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:14.965731 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:14.965703    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:16.525888 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:16.525857    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:16.579168 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:16.579146    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:18.068598 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:18.068569    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:18.119690 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:18.119660    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:19.579850 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:19.579822    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:19.624221 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:19.624195    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:21.132061 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:21.132031    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:21.178332 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:21.178311    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:22.649446 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:22.649419    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:22.697023 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:22.696999    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:24.151913 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:24.151886    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:24.207535 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:24.207506    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:25.691427 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:25.691399    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:25.739263 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:25.739232    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:27.252225 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:27.252201    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:27.304529 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:27.304495    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:28.856484 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:28.856458    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-54c789bdc6-22wcp_a339f3e3-5a39-418e-96ba-1436872939a0/istio-proxy/0.log"
Jul 06 22:23:28.918117 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:28.918091    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh_ad17c5df-6533-4378-b417-259efe93ba3c/main/0.log"
Jul 06 22:23:39.092667 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:39.092641    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:40.400615 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:40.400582    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:41.759640 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:41.759602    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:43.074710 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:43.074675    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:44.424958 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:44.424933    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:45.751026 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:45.750999    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:47.048842 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:47.048796    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:48.434385 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:48.434358    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:49.728275 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:49.728245    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:51.057183 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:51.057148    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:52.363470 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:52.363445    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:23:58.206396 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:58.206354    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dnd82_1cd6de51-6683-4b89-b584-0cb22a3b8833/global-pull-secret-syncer/0.log"
Jul 06 22:23:58.396543 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:58.396519    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zbdwk_8f090afa-8dff-4167-b753-523344760615/konnectivity-agent/0.log"
Jul 06 22:23:58.420395 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:23:58.420369    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-45.ec2.internal_3e825bc10f7326db6e76e5c99478ee31/haproxy/0.log"
Jul 06 22:24:03.505424 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:03.505349    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-fsfzc_7210fbb3-0495-4e85-bf34-1e768c0b0719/limitador/0.log"
Jul 06 22:24:05.090465 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.090437    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-v2h4j_343a6f78-6839-47a4-9a6c-4f71f52aa9f0/kube-state-metrics/0.log"
Jul 06 22:24:05.119897 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.119873    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-v2h4j_343a6f78-6839-47a4-9a6c-4f71f52aa9f0/kube-rbac-proxy-main/0.log"
Jul 06 22:24:05.142507 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.142485    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-v2h4j_343a6f78-6839-47a4-9a6c-4f71f52aa9f0/kube-rbac-proxy-self/0.log"
Jul 06 22:24:05.319977 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.319949    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sldgn_bcc61e35-edac-4bcf-a81a-80dd8ea0a029/node-exporter/0.log"
Jul 06 22:24:05.341830 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.341738    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sldgn_bcc61e35-edac-4bcf-a81a-80dd8ea0a029/kube-rbac-proxy/0.log"
Jul 06 22:24:05.365283 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.365266    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sldgn_bcc61e35-edac-4bcf-a81a-80dd8ea0a029/init-textfile/0.log"
Jul 06 22:24:05.497387 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.497362    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-45gjw_69c4ee10-5ebb-44d8-9552-da4996f18848/kube-rbac-proxy-main/0.log"
Jul 06 22:24:05.524816 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.524795    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-45gjw_69c4ee10-5ebb-44d8-9552-da4996f18848/kube-rbac-proxy-self/0.log"
Jul 06 22:24:05.564038 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.564015    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-45gjw_69c4ee10-5ebb-44d8-9552-da4996f18848/openshift-state-metrics/0.log"
Jul 06 22:24:05.633742 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.633666    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d65eec04-be37-4369-95ad-74eea6e07f53/prometheus/0.log"
Jul 06 22:24:05.651538 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.651516    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d65eec04-be37-4369-95ad-74eea6e07f53/config-reloader/0.log"
Jul 06 22:24:05.703138 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.703111    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d65eec04-be37-4369-95ad-74eea6e07f53/thanos-sidecar/0.log"
Jul 06 22:24:05.767647 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.767621    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d65eec04-be37-4369-95ad-74eea6e07f53/kube-rbac-proxy-web/0.log"
Jul 06 22:24:05.799649 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.799613    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d65eec04-be37-4369-95ad-74eea6e07f53/kube-rbac-proxy/0.log"
Jul 06 22:24:05.840490 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.840467    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d65eec04-be37-4369-95ad-74eea6e07f53/kube-rbac-proxy-thanos/0.log"
Jul 06 22:24:05.869985 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.869955    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d65eec04-be37-4369-95ad-74eea6e07f53/init-config-reloader/0.log"
Jul 06 22:24:05.915853 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.915778    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-mnpck_acb32073-6101-4721-a335-582734e6e8bc/prometheus-operator/0.log"
Jul 06 22:24:05.940829 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.940790    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-mnpck_acb32073-6101-4721-a335-582734e6e8bc/kube-rbac-proxy/0.log"
Jul 06 22:24:05.973339 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:05.973320    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-djgk4_370f67b9-b322-455b-b7df-099c1b5750f5/prometheus-operator-admission-webhook/0.log"
Jul 06 22:24:08.334622 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.334589    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"]
Jul 06 22:24:08.335029 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.334927    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e62cd4d-81a3-4e0c-b892-9dbc91563a19" containerName="main"
Jul 06 22:24:08.335029 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.334939    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e62cd4d-81a3-4e0c-b892-9dbc91563a19" containerName="main"
Jul 06 22:24:08.335029 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.335004    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e62cd4d-81a3-4e0c-b892-9dbc91563a19" containerName="main"
Jul 06 22:24:08.338260 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.338241    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.340871 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.340844    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hm2m\"/\"kube-root-ca.crt\""
Jul 06 22:24:08.341889 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.341868    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8hm2m\"/\"default-dockercfg-k7bqq\""
Jul 06 22:24:08.341993 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.341913    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hm2m\"/\"openshift-service-ca.crt\""
Jul 06 22:24:08.348719 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.348696    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"]
Jul 06 22:24:08.385233 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.385204    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-sys\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.385348 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.385238    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-proc\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.385348 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.385266    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-lib-modules\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.385421 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.385342    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-podres\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.385421 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.385380    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzj4\" (UniqueName: \"kubernetes.io/projected/bb2990ba-c4c0-4cf7-9436-3c62c3429990-kube-api-access-7pzj4\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.486659 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.486633    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-sys\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.486804 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.486665    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-proc\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.486804 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.486711    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-lib-modules\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.486804 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.486764    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-sys\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.486804 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.486776    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-proc\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.487070 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.486839    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-podres\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.487070 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.486883    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-lib-modules\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.487070 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.486901    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzj4\" (UniqueName: \"kubernetes.io/projected/bb2990ba-c4c0-4cf7-9436-3c62c3429990-kube-api-access-7pzj4\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.487070 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.486923    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb2990ba-c4c0-4cf7-9436-3c62c3429990-podres\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.495865 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.495833    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzj4\" (UniqueName: \"kubernetes.io/projected/bb2990ba-c4c0-4cf7-9436-3c62c3429990-kube-api-access-7pzj4\") pod \"perf-node-gather-daemonset-p8dxp\" (UID: \"bb2990ba-c4c0-4cf7-9436-3c62c3429990\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.539800 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.539783    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69f57c9ff6-b62bc_4372b006-c5df-4dd2-950f-608658ed33fd/console/0.log"
Jul 06 22:24:08.579415 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.579387    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-qkh89_6c4dcd10-1f2f-4405-805a-9694a2122c6f/download-server/0.log"
Jul 06 22:24:08.648895 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.648836    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:08.776464 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:08.776443    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"]
Jul 06 22:24:08.778731 ip-10-0-131-45 kubenswrapper[2550]: W0706 22:24:08.778706    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbb2990ba_c4c0_4cf7_9436_3c62c3429990.slice/crio-3923d549dd19ae0f4b8560b056157af76f545cdcd6d4e94f35a94df0d0c70ed2 WatchSource:0}: Error finding container 3923d549dd19ae0f4b8560b056157af76f545cdcd6d4e94f35a94df0d0c70ed2: Status 404 returned error can't find the container with id 3923d549dd19ae0f4b8560b056157af76f545cdcd6d4e94f35a94df0d0c70ed2
Jul 06 22:24:09.042789 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:09.042694    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp" event={"ID":"bb2990ba-c4c0-4cf7-9436-3c62c3429990","Type":"ContainerStarted","Data":"55f77624258ce47d62afa275051f94caeff995dffdf049e28fc72f24c0b79122"}
Jul 06 22:24:09.042789 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:09.042744    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp" event={"ID":"bb2990ba-c4c0-4cf7-9436-3c62c3429990","Type":"ContainerStarted","Data":"3923d549dd19ae0f4b8560b056157af76f545cdcd6d4e94f35a94df0d0c70ed2"}
Jul 06 22:24:09.042789 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:09.042764    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:09.060988 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:09.060946    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp" podStartSLOduration=1.060932366 podStartE2EDuration="1.060932366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:09.060359555 +0000 UTC m=+6562.001663394" watchObservedRunningTime="2026-07-06 22:24:09.060932366 +0000 UTC m=+6562.002236213"
Jul 06 22:24:10.240119 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:10.240091    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nb4m7_3c6624a7-c191-44b2-806f-c1a04a48402e/dns/0.log"
Jul 06 22:24:10.262745 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:10.262725    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nb4m7_3c6624a7-c191-44b2-806f-c1a04a48402e/kube-rbac-proxy/0.log"
Jul 06 22:24:10.378675 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:10.378656    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fp74t_5eef1e47-1131-4820-b054-5ad627772e0f/dns-node-resolver/0.log"
Jul 06 22:24:11.110681 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:11.110656    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mqszb_6fe996a8-9939-4be3-a9d2-a574dbc4ce24/node-ca/0.log"
Jul 06 22:24:13.766232 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:13.766198    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-57cbl_ec945c9f-5716-49f2-8dd7-0347fe6275dd/serve-healthcheck-canary/0.log"
Jul 06 22:24:14.339348 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:14.339320    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4ct7m_1f3b2b13-dcaa-4ce6-8e00-201340f91ed0/kube-rbac-proxy/0.log"
Jul 06 22:24:14.366007 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:14.365983    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4ct7m_1f3b2b13-dcaa-4ce6-8e00-201340f91ed0/exporter/0.log"
Jul 06 22:24:14.391320 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:14.391297    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4ct7m_1f3b2b13-dcaa-4ce6-8e00-201340f91ed0/extractor/0.log"
Jul 06 22:24:15.056447 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:15.056421    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-p8dxp"
Jul 06 22:24:17.723927 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:17.723897    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7845fd6587-tfltn_9a7f455f-ba2e-4ee7-a65c-88607b1e9eff/manager/0.log"
Jul 06 22:24:17.782844 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:17.782781    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-r25kc_102c9d2e-fb80-4a6c-aedc-5516259870f3/openshift-lws-operator/0.log"
Jul 06 22:24:18.433159 ip-10-0-131-45 kubenswrapper[2550]: I0706 22:24:18.433131    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-555fd59bd5-82brg_9b9f8294-9be8-4620-a54f-c1f048188734/manager/0.log"